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ABSTRACT 



In a proposed live-state demon at rati on, the four 
states or Arkansas, Minnesota, Mississippi and Nevada implemented a 
local system cf directing evaluations of voca t ional/tech nical 
education programs in public schools. The emphasis was on Loth the 
training of local leaders by state leaders and direction of 
evaluation programs by local leaders. Using the results of cwo 
previous projects, this particular project generalizes t he findings 
to other states. The methods used to aeguire and disseminate 
information were workshops, project reporting meetings, final report 
and project evaluation meetings, and o.i-site visits. The results of 
the project indicate that most of the evaluation procedures are 
applicable to other states, that some additional procedures were 
developed by individual states, and that program evaluation 
competencies were developed in the personnel involved in the project. 
It was recommended that states meeting the criteria for selection of 
states usea in this study try out the systems* Further study and a 
nationwide training program were also recommended. (GER) 
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I- SUMMARY 



The Problem 



The basic problem dealt with in this study was the further try-out 
and demonstration of a system for locally directing evaluations of 
public school programs of vocational - technical education, and the further 
try-out and/or development and demons tra tion of a procedure for state 
leaders to provide training of local leaders in such an effort. The 
system tried out and demonstrated was the major outcome of two previous 
Michigan projects involving locally directed evaluations in 13 school 
sys terns . 

The steps in the system relate to administrative support, and 
faculty and citizen involvement on an extensive and an intensive basis, 
with focus on outcomes, and a review of local programs in the light of 
manpower needs and student and citizen interests and needs. 

Scope of the Study 

As implied in the title, the try-out and demons trat ion was planned 
Lo be done in five states. Actually, just four were iir; i ved , due to 
staffing difficulties in the fifth state. Hereafter, this study is 
referred to as the "Mul t i -State Project.' 1 The four states involved were 
Arkansas, Minnesota, Mississippi, and Nevada. In each of these a state 
project leader was designated who served for a period of two years. 

Each leader selected five schools to cooperate with him. Eleven of these 
were local school districts, seven were county districts and two were 
post-secondary institutions. Eace of these schools named a local leader- 
ship team of tv/o faculty members. Thus, 20 schools systems, with 40 
local leaders in them were most deeply involved. In addition, however, 
291 local staff members, who served on local staff committees, and 
roughly 473 citizens serving on advisory committees were directly 
involved. Unnumbered other faculty members in these schools and citizens 
in the communities, counties or areas served also were indirectly in- 
volved . 

The state projects, following orientation meetings in November 
and December 1967, began with a workshop for state leaders in March, 

1968. The work of the state leaders was completed in January, 1970. 

The participation on the part of local personnel began in the spring 
of 1968, and extended through the fall of 1969, or roughly 20 months. 

Activities for disseminating information about the project were 
carried on in the communities and states involved by the appropriate 
leaders during this period. Further disseminai ion was carried on by the 
chief investigator, particularly in 1970. During this third year of the 
project three workshops were conducted in different geographical 
locations, for a total of 132 state, area and local leaders in the field 
of vocational /technical education, assisted by state project leaders and 
12 local project personnel. 
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Objectives of the Study 



The major objectives of the study were stated as follows: 

- To determine the feasibility and general izabili cy of a state 
procedure of assisting local school district leadership to 
use the evaluation system developed in Michigan, 

- To discover and/or develop new or improved procedures in a state 
system for local evaluations. 

- To assist in development of state and local leadership competen- 
cies in evaluation of local programs of vocational and technical 
education, including creation of understanding of the values of 
local involvement . 



Advisory councils were used to select the states, to select schools 
within states, and to provide advice on the local project in cooperating 
schools. Criteria were established for selecting states, and schools 
within each state. A workshop, two projecc reporting meetings, and 
final report and project evaluation meetings were held with all state 
leaders. Each leader conducted a workshop, two, and in one case three 
project review meetings and a final report and evaluation meeting. 

On-site visits were made for orientation, monitoring and consultation 
purposes. Instructional materials, likewise, were provided at all levels. 
All cooperating schools were expected to uniformly plan and conduct 
certain activities, which were conceived of as a part of the ’’system", 
but they also were encouraged to try out other activities. Quarterly 
reports were made by local leaders to each state project leader, and 
by them to the chief investigator. 

The procedure for evaluating the system was begun by establishing 
a series of criterion questions for each of the three major objectives. 

For each of these questions the evidences to look for were listed, 
along with sources of factual information and findings contributing to 
these evidences. These were considered by state project leaders and 
the chief investigator, assisted by a consultant on evaluation. Pro- 
cedures for evaluating the regional workshops was developed separately, 
but in a similar manner. 



Participating schools were mostly smaller or medium In size - 
1,267 to 6,759 in K-12 enrollments - with only tw> in the 28,000 range. 
All had programs of vocational education in several fields, with pre- 
dominance In home economics, business, Industrial, and agriculture. All 
met criteria established by state leaders and their advisory comm! t tee j 
for selection* 

The leadership teams appointed in these schools accepted the mini- 
sum activities, and i3 of the schools carried them to completion. In 
addition, 19 other activities contributing to the evaluation effort 
were engaged in by from one to six schools. The activity not universally 



Methods Used 



Results Obtained 
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accomplished to the extent desired by state leaders was that of formulat- 
ing and stating program objectives. This may reflect a weakness in the 
system or in the training provided by state or local leaders. Those 
activities accomplished to a higher degree included sta r f involvement 
in all schools, use of citizens' committees in 17 scht >ls , and condi cling 
follow-up surveys in 12 secondary systems and in .wo post- c nd ry 
systems. All schools studied existing programs in t ne lig t of manpower 
needs and parent and student intei es s and \ rceived needs. This was 
done in connection with follow-up surveys; ;tudy of student interests 
through OVIS in 12 sc,>ols; parent surveys in nine schools; employer 
interviews in eight chools; occupational surveys in four schools, and 
in a variety of other acti\Lties. 

Developments and changes reported as occurring during the local 
ovoject year were improved statements of philosophy, and improver, under- 
standing and coordination of faculty, administration and citizens. 

Specific changes included addition of specialized vocational courses, 
related courses and "exemplary programs", addition of class sections for 
existing courses, establishing the position of local director of 
vocational education, placement bureaus, new equipment and many others. 
Recommendations from these studies emphasized addiding of courses to the 
school programs, staff and facilities development, counsel } tv; , and 
guidance, and a variety of other pertinent changes. 

Ail 20 schools reported that there were plans for continued directing 
of evaluations locally, although not necessarily every year with alL the 
activities that were engaged in during the project year. Staff and 
citizen reactions revealed through depth interviews and a check list 
indicated positive support for the activities in which they engaged . 

On the part of the state leaders and their state projects, all 
accepted the Multi-State Project objectives, and stated some object We* 
in addition for their own states. A plan for each state was prepared 
by state leaders. Their basic procedure^ were quite uniform and 1 ive 
been described in more detail elsewhere in this report. All rf nu pro- 
jects were carried to completion despite major staff changes in two 
states and a minor change in another state. All condu ;lnd communication 
and dissemination activities in their states, assisted in revising the 
manual on local program evaluation, and cooperatively assembled and 
analyzed information pertaining to the basic criterion questions and in 
drawing conclusions. All were also involved in one or more of the work 
shops conducted as a part of the dissemination phase. Considerable 
activity in training of local leaders in evaluation procedures has been 
engaged in subsequently by these state leaders in three states. 

Highlights of the Findings, Significance, and Implications 

Following are a few of the highlights of the findings. 

1. The replicatioii of the system for locally directed evaluation 
has been established as feasible and gener alizable in smaller 
and medium-size school systems. (Objective No. 1) 
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2. 



It appears that any state meeting the criteria used in selecting 
the states in this project could replicate the system of state 
assistance and training for local leaders in conducting evaluat- 
ions In school systems comparable to those in the four coopera- 
ting states. (Objective No. 1) 

3. New or improved practices, developed and/or tried out in one or 
more cooperating states are reflected in the revised manual, 

in the dissemination workshops held, and in plans for future 
work in these states. (Objectives No. 2) 

4. Nine individuals in state leadership positions and eight 
assistants developed competencies in evaluation of local 
programs and in training local leaders in practice and pro- 
cedures of evaluation of local programs through the two-year 
participation and assistance in the dissemination workshops. 

In addition, 73 persons in positions of state leadership in 
37 states developed competencies in these workshops. This 
project, therefore, did assist in developing such competencies. 
(Objective No. 3) 



Recommendations 

1. States ti at meet the criteria used in this study cor selecting 
states should try out the systems demonstrated in the present 
study. Leadership from several sources at the state level, 
along with financial resources, should be utilized to conduct 
programs of local leader training which would be like those 
conducted in this project, but improved through application of 
improvements developed and tried out. Further assistance 
should be given in improvement of some instruments and develop- 
ment of o ther s . 

2. A nationwide program of training in local program evaluation 
procedures and for assisting with locally directed evaluations 
will be needed, primarily for persons at the StaLe level of 
leadership, but also for administrators and teachers in local 
and area institutions. 

3. Further study should be given to: (a) relating local evalua- 

tions and statewide evaluations to each other; (b) developing^ 
trying out and/or demonstrating new, or improved practices; 

(c) assisting local leaders to forroulate program objectives; 

(d) adapting the system to self-directed evaluation of area 
vocational/technical school and community college programs; 
and (e) analysis of manpower needs and interpretation for 
curriculum and program revision. 
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II INTRODUCTION - BACKGROUND 



Overview 

This report has thus far presented a brief summary of the Multi- 
State Project as planned and carried out. This section introduces the 
general design and provides the background of the study. Included in 
this section are the nature of the study and of others either antecedent, 
or concurrently related to it; the scope; the general objectives; the 
limits; and the significance of the project. In the section that follows 
this the methods of conducting the study will be presented. The main 
body of the report is a condensation of the results, with emphasis on 
activities and results in the four cooperating states: Arkansas, 

Minnesota, Mississippi and Nevada. This includes all three phases; 

(1) orientation and preparation of, and planning by State leaders and 
local leadership teams and staff committees; (2) carrying out and re- 
porting on accomplishment and determining attainment of the general 
objectives; and (3) dissemina t ion of findings and results. Following 
these, conclusions are drawn and recommendations are made. 

delated Studies 



A brief look at related studies provides additional information to 
supplement other background information. These may be identified in 
two greups: those leading rather directly to the present study f 

and those that are related, and which contributed in a significant way 
to it. 



t s udy in 19:8 was made by Byrain of factors associated with develop- 
ment o. c comprehensive programs of vocational education In 20 Michigan 
public schools. Among the findings were that state-level leadership and 
use of citizens' committees had not been utilized; and that some schools 
had conducted follow-up surveys. (3) Subsequently a comprehensive, 
state-wide evaluation of vocational education in Michigan was ade. 

Among the recommendations growing out of this s tudy, complr t d in 1963, 
were these that help to point up the need for attention to local direc- 
tion of program evaluation. 

"...that the Division of Vocational Education take the leader- 
ship in developing criteria that local communities may use 
as guide-lines in evaluating their vocational curricula..." 

"...that the Division of Vocational Education encourage local 
communities to establish long-range plans for program develop- 
ment and program improvement ... " 

"...that local districts regularize the follow-up studies of 
employment -bound youth, dropouts in the vocational education 
program, and graduates of vocational education programs to 
determine labor market behavior." 
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. . that . advisory committees composed of appropriate repre- 
sentation should be required at all levels of operation." (1) 



The Michigan Department of Education, by way of partially implemen- 
ting these recommendations, provided partial support for the first 
Michigan project cn evaluation of local programs, started in 1963. The 
purpose vas to study, develop, and try out local evaluation procedures. 
This was done in three representative school systems. Upon completion 
of this study in 1965 it was recommended that the project be extended 
to other schools in the state to; further try out and improve upon 
procedures; to study the role of the local evaluation leader and of 
consultants; and to test the hypothesis that advisory committees on 
vocational education are essential to program evaluation. (4) 

This recommendation was implemented in 1966 through a project 
supported by the U.S. Office of Education and M.S.U., in which 10 
Michigan schools cooperated. It was recommended at the close of this 
project that the "system” that was developed be tried out in its same 
form, or be modified to suit needs , in other states. (5) The proposal 
b;sed on this recommendation led to the present study. 

During the conduct of these last three projects a number of other 
studies were made or reported in other states which had a bearing on 
the way in which the three local evaluation projects were conducted and 
the foci of various emphases in them. A report by Hamlin, (10) was 
helpful in selecting and fitting together local staff and citizen activi- 
ties that might be called a system. A report by Burt ,(2) one by 
Wenrich (20) and a later report by Hamlin (11) contributed to the 
citizen involvement phase. Studies by Sharp, (17) Eninger (9) and 
Pucel (16) and guidelines by the Michigan Department of Education (13) 
and others contributed to procedures for the follow up. The research 
reports by Starr(18) and Moss (14) provided guidance in development 
objectives, and in developing procedures for identifying and retrieving 
pertinent information. A survey by Cromer (7) aided in study of pro- 
gram needs. Other studies having direct or indirect bearing arc listed 
in the bibliography. 

There have been many reports relating in some was to local program 
evaluation. Nearly all, however, de^l with such evaluations as being 
state designed and/or state directed. They have not emphasized involve- 
ment of local educators and citizens and/or they dealt mainly or 
exclusively with input or process, rather than output or outcomes, hence 
the justification for the present study. 



Nature of the Study 



The basic problem dealt with in this project was the further try- 
out and demonstration of a system for locally directing evaluations 
of public school programs of vocatlonel/technlcal education, and the 
further try-out and/or development and demonstration a procedure 
for state leaders to piov^ training of local leaders in such an 
effort. The findings ftc wo previous Michigan projects on locally 
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directed evaluations involving a cotal cf 13 schools were used. These 
findings were in terms of workable or effective general procedures. The 
general aim of the present project was to determine the extent of their 
general izability to other states, and to demonstrate how state leader- 
ship In this endeavor may be prepared and utilized. 

Emphasis on local faculty and citizen involvement are described in 
terms of relevant activities carried out it the 20 cooperating schools. 
Emphasis on output or outcomes of vocational education programs is made 
manifest in the descriptions of the work of local cooperating school 
systems in developing program objectives, in conducting follow-up 
survey and in other ways. Concurrently, the leadership training in 
evaluation for state project and local project leadership is described. 

Detailed quantitative findings in cooperating schools and states 
are not given in order to avoid undue bulk in this report. Rather, 
because of their importance to the objectives of the project, qualitative 
information is emphasized, along with details regarding procedures and 
methods used by the leaders in bringing about a rial of the replic- 
ability or gener altzabi 1 i ty of the "system 1 . 1 

Scope Of The Report 

The scope of this report covers activities beginning in early fall 
of 1967, and continuity through 1970 m the part of the chief investigator 
and his assistant. It also embraces the work of the State project leaders 
which extended through 1968 and 1969. The work of local project leaders 
extended from spring, 1968 to fall, 1969 and is also included in the 
description. Certain local leaders' and State leaders' work as consul- 
tants at regional workshops in the die iemina t ion phase took place in 1970. 

The report deals mainly with vocational education programs in 
local district schools, because the systematic approach had been developed 
with these schools in the earlier Michigan project. County unit systems 
and area secondary vocational schools were not available -In Michigan at 
that time. Where county unit systems, area vocational r bools and post- 
secondary programs were included, this report describes what took place 
in the evaluation of programs of vocational education in them. 

Rationale Of The Study 
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The recent concern on the part of public schoo. systems in this 
country with evaluation of local programs of vocational and technical 
education arises from several sources. Citizens and school people have 
become increasingly aware of the role of the schools in the occupational 
education of youth and adults. Attention customarily given to separate 
or distinct programs in the traditional fields of vocational education is 
being accompanied by concern for the contribution of all specialized 
curricula, and of the school as a whole to occupational preparatijn. The 
concern for accountability of resources, used for programs of vocational 
education as expressed at National and Stati levels is also made manifest 
at the local level. 
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Although many studies have been conducted, and reports have been 
made dealing with program evaluation most of the studies have not b< en 
designed so as to give attention to locally derived oLjectives or to 
involvement of local educators and citizens in activities to bring about 
program evaluations. The emphasis in these studies has been placed 
largely on state and/or national objectives and the appraisal of programs 
toward attainment of these objectives. A cursory examination of these 
would show that only token consideration has been given to a system for 
local program evaluation that uses self study and the identification of 
goals and outcomes, with emphasis on involvement of staff and citizens, 
while utilizing consultative assistance from experienced vocational/ 
technical educators. 

The present project provides several points of emphasis not present 
to any large degree in other completed projects dealing with evaluation. 
In the first place, the emphasis is placed on involvement. Local 
public school programs of vocational and technical education have been, 
and will continue to be evaluated by those involved in or responsible 
for the programs and by those affected by them. Citizens, including 
taxpayers, parents, and clientele of courses, do evaluate such programs. 
They may or may not evaluate them fairly, but they do it, nonetheless. 

The interests of the professional people are or should be in consonance 
with these. 

Citizens and local professional people are primarily interested in 
whether the outcomes are commensurate with their desires and with the 
resources supporting the programs. They typically have only a token 
interest in comparing their programs with those in ocher schools, with 
theoretical standards, or with standards established by outside agencies. 
The kind of checking of ways and means utilized in local programs, as 
is done by typical accreditation bodies ov by agencies of government 
has value, but was not the focal point in the present project, even 
though the ’’evaluative criteria" used by some were examined. 

The elements of an effective, locally directed evaluation identi- 
fied in previous projects were administrative approval and commitment; 
available qualified and well prepared local leaders; development of local 
staff competencies in evaluation procedures; provision of leader and 
staff time for evaluative activities; effective use of staff committees 
and citizens 1 advisory committees* use of sound research methods; and 
adequate consultative services. These elements formed the basis for 
the ’‘minimum activities" to be required of state project leaders and 
by them of local leadership teams. They grew out of the experiences 
and results of the Michigan projects. Flexibility In the conduct of 
state and local projects was desirable, and was provided for so as to 
help realize objective No, 2 of the project. 

Limits of the Study 

This study was limited to the four cooperating states and to the 
state level leadership in each state; to the five schools within each 
state and the 20 local/area leaders in these schools; to materials 
^ ared in furtherance of the study; and to th ‘ three dlss emi.nat ion 
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workshops and other activities in the third phase involving 132 educators 
in 43 states . 

It was limited to program evaluation and did not attempt to assess 
or measure achievement or to develop or test instruments for such 
measurement. The term ''program" does not connote limitation to Federally 
reimbursable programs under National Vocational Education Acts. Indeed, 
the only limits in this regard were those set locally, and thus permitted 
and encouraged consideration of the school's total contribution to 
occupational education of youth and adults. 

Significance of the Study 

The major contribution of this study is seen as the systematic 
approach to locally directed program evaluation that was tried out and 
demonstrated. Those states and sta^e leaders interested in such evalua- 
tions should, as a result, now have greater confidence in applying or 
following the steps in the system. Local and area schools should be in 
a better position to evaluate their programs leading to improved local 
program planning. 

A secondary, but nonetheless important outcome of the study, is a 
cadre of well prepared educators, (1) at the state level for initiating 
and conducting training programs for local and area leaders, as well as 
dissemating information to leaders in other states; and (2) at the local 
or area level to continue with effective locally directed evaluations. 

The competencies they have developed could be utilized by their peers in 
comparable institutions. 

The third phase of the study, dealing with dissemination is viewed 
as contributing primarily to the third objective. The significance 
of this phase will come to light only after those who participated in, 
and were affected by these activities are dernonst? atably able to follow 
and implement procedures in their own states or institutions. lh 4r . 
further testing period Is beyond the time span covered by the p. esent 
s tudy . 

Another product of this study is the further revision and publication 
of the manual for local and state leaders previously developed and 
based on the preceding Michigan projects. This should be a valuable 
instructional resource in the necessary training programs. 

Finally, and by no treans incidentally, there is presented here a 
major approach to educational evaluation and planning which should pro- 
mote the furtherance of goals of democracy In education. It has been 
assumed in this study that local programs of vocational education should 
be evaluated \>y those responsible for them - the administrators and 
teachers - and by those affected by them, or who are th * beneficiaries 
of such programs - the citizens, employers, and students. 




13 



-9- 



Since the provision for such Involvement is not generally provided 
for by accrediting agencies nor by agencies checking on compliance vith 
licensure standards, or statewide reimbursed vocational education, cr 
making evaluations of statewide or nationwide programs, the possible 
significance of the findings of this study is anticipated. 
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III. METHODS OF CARRYING OUT THE STUDY 



General Procedure 

The general procedure followed in thi r project was to replicate and 
tiy out a system of local direction of evaluations of programs of voc- 
ational education through state assistance and training of local personnel 
to be involved. Specifically, the three purposes or objectives of the 
project were stated as: 

-To determine the feasibility and gener a li zabi li t y of a 
state procedure for assisting local school district 
leadership to use the evaluation system developed in 
Michigan . 

-To discover and/or develop new or improved procedures in 
a state system for local evaluations. 

-To assist in development of state and local leadership com- 
petencies in evaluation of local programs of vocational 
and technical education, including an understanding of the 
values of local involvement. 

The replication involved the selection of states, in which a local 
leader was then appointed, and a consortium arrangement developed. It 
involved preparatory orientation, training, consultative service, and 
instructional resources provided to the cooperating states. The sys- 
tematic approach tried out in Michigan was the model used for replication 
and try out. (5) This means that each state leader used an advisory 
committee, selected representative school systems, provided orientation, 
consultation and monitoring services, and assisted the local leadership 
teams to identify and report pertinent results of each evaluation activ- 
ity. 



The evaluation of the system involved the determination of the 
attainment of the three purposes (objectives) stated above. In c'-dcr 
to determine attainment of tne first objective, the feasibility and 
generalizabi li ty of the system for local programs and state programs 
were defined. Feasibility was defined as the extent t n which the eval- 
uation system was followed in the schools and the extent to which the 
methods of providing state leadership were followed. Genera lizabi li ty 
was defined as the extent to which activities and plans within each co- 
operating state were comparable and the extent to which the system for 
providing state leadership to the local systems were similar to one an- 
other and to that of the Michigan project. Criterion questions were 
then set up. These are as follows: 

Gener f 'izability and Feasibility - Local Programs 

Were the minimum activities attempted in each school? 

Weri the activities attempted actually carried out? 

Did cooperating schools vary as to size, type of organization, 
type of prceram and/or objectives? 

Were variations of schools related to evidences on questions 
1 & 27 
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5. Was faculty involved to a high degree? 

6. Were citizens highly involved in each school during the pro- 
ject? 

Effectiveness of System-l o cal Programs 

1. Have changes in local programs b en planned and/or made, either 
directly or indirectly as a result of the project? 

2. Has a change in attitude toward, or understanding of vocational 
education taken place in the faculty of each school during the 
project? 

3. Has a change in attitude toward, and/or understanding of the 
vocational education program of the school by the citizens in- 
volved taken place during the project? 

4. Will s elf -ini tiat ed local evaluation be conducted in future 
years on a regular or continuing basis in the cooperating 
schools? 

5. Were local citizens' av*d educators' reactions to local evaluation 
practices favorable? 

Generalizabi li t y of State Systems 

It Were objectives of state projects similar? 

2. Were activities in conducting state projects similar? 

3. Were state project plans and activities adapted to differences 
among the states? 

Sources of evidence on these questions were identified, and information 
needed to answer them was listed. This information was to be incorporated 
into local project leaders' and state project leaders' final reports. 

The attainment of Objective No. 2 was planned to be checked period- 
ically in visits by the investigator to monitor projects in cooperating 
states, and in the final meeting of state project leaders. The attainment 
of Objective No. 3 was not planned to be formally analyzed. The project 
as a whole could be conceived of as a major leadership training activity. 
It was planned to note these training activities throughout the project, 
and at the close. 



Selection of States, and Personnel Involved 

The plan for selection of cooperating states was put into effect in 
September, 1967. Announcement of the availability of the project was 
mailed to 43 state directors of vocational education. Later that month 
at a national institute on teacher education the project was discussed 
individually with interested teacher educators from 10 states. Written 
and oral communications expressing interest in participating in the pro- 
ject came from 17 states. Follow-up letters and additional information 
were sent to these states in the form of an abstract of the project pro- 
posal. 



On October 25, 1967, an advisory committee of rine leaders in the 
field was brought together. The membership consis ed of five from Michi- 
gan and four from other a:ates (See Appendix A). This meeting was devoted 
O reviewing the criteria for selection of cooperating states, to dis- 




18 



- 12 - 



I 



cussing information about the states, and to giving advice about the 
selection. The following criteria were applied in the selection of the 
states: 

1. The state has a sufficient number of schools of a size appro- 
priate for conducting comprehensive progx'ams of vocational- 
technical education from which to select cooperating schools. 

2. The state does not already have a system in operation of locally 
directed program evaluation on a state-wide basis. 

3. There is an endorsement of the project by the state department 
of education, and the prospect of sufficient resources to be 
committed so as to assure that the project is likely to be 
carried out as planned. 

4. The agency or the institution has the staff capability to pro- 
vide necessary leadership. 

It was decided that there should not be a strong emphasis on geo- 
graphical location of the states. Geographical representativeness was 
considered, but the above criteria were considered more important. The 
five states tentatively selected and notified of their selection were 
Arkansas, Minnesota, Mississippi, Nevada and Virginia. 

Visits were made to the first three of these states in November to 
provide additional information and to discuss important aspects of the 
project. Similar meetings were held in December with representatives of 
Nevada and of Virginia at the site of a national meeting. Immediately 
following these, an orientation meeting was held for representatives from 
all five states. These included in each case the proposed state project 
leader . 



On December 13 word was received from Virginia that it could not 
participate because of inability to provide time for a state project- 
leader, Contacts were made with three other states to ascertain possi- 
ble interest in being the fifth state, with either delayed or negative 
results. Further effort toward recruiting a fifth state was di scor'iinued 
because of need to get the project under way as soon as por~jhl i , because 
of delay by U.S.O.E. in the am Tuncement regarding funding of the project, 
and because of the drastically reduced funds for the fiist contract year. 
The project thereafter was referred to as the "Multi cate Project," 
and this designation will be used in the remainder of this report. 



The four educators appointed as state leaders in the respective 
states were: Mr. Edward Cordisco, Assistant Director of Vocational Ed- 

ucation, Nevada; Dr, R, Paul Marvin, Professor of Agricultural Education, 
University of Minnesota; Dr. Robert E. Norton, Assistant Professor of 
Vocational Education, University of Arkansas; and Dr. James E, Wall, 
Director of Research Coordination Unit, Social Science Research Center, 
Mississippi State University. Floyd McKinney served as assistant on the 
Multi -State Project from September 1, 1967, to May 31, 1969; Marvin 
Robertson from June 1, 1969 to May 31, 1970; and Gordon Ferguson from 
June 1, 1970 to December 31, 1970. 




Leadership changed occurred in Nevada, Minnesota, and Mississippi. 
Increasing demands on the time of the limited state staff in Nevada 
forced that state department to sub-contract for the proj ect -dir ec ting 
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services of Dr. Howard Christensen, Professor of Agricultural Education, 
University of Nevada from March 1969 to January 31, 1970. Mr. Edward 
Hartog, as Istant on the Minnesota project, served as acting state leader 
at the University of Minnesota from September 1, 1968 to March 1, 1969 
while Dr. Paul Marvin was on a leave of absence. Dr. James Shill, 
starting as assistant on the project in Mississippi, became co-director 
of that project during the latter part of 1968. Other assistants to the 
state project leaders were George Brooker, Don Noble, and Buck Lowery in 
Arkansas, George Copa in Minnesota, and Denis Graham and Ivan Lee in 
Nevada. Thus, a total of 17 persons in state level positions were involved 
during all or a part of the first two phases of the project. 



Selection of School Systems in Each State 

The following criteria were used in the four cooperating states in 
selection of school systems with which to work: 

-The administrators and faculty appear to have a commitment to, and 
give support to a local program evaluation effort. 

-The school system has vocational/technical education programs in 
at least three (four) occupational fields. 

-All occupational fields are represented in one or more schools, 

-The school systems are located in somewhat representative geo- 
graphic areas of the state. 

-The school systems vary as to size of student enrollment. 

-The schools are recommended by a state advisory committee for the 



In Mississippi a variety of systems was desired, such as: 

-A municipal separate district school system 
-A county unit system 
-A consolidated school system 
-A junior college jistrict system 

In three states five school systems were selected. In Arkansas six 
were selected but one subsequently withdrew because of lack of time for the 
local leader, Each system was asked to appoint a leader and an associate. 
Thus, 20 schools and 40 members of local leadership teams were involved. 

In Minnesota it was desired to have at least one to be an area voca- 
tional school. T n that project the Secondary school, administered joint- 
ly with the post -secondary area vocational school was not included. One 
of the other Minnesota systems also had an area school, but that area 
school was not a part of the project. In Nevada all five systems were 
county unit administered, but had locally-programmed secondary schools. 

The 20 cooperating schools resulting from this selection process are as 
follows: 



Replication of the System 



project 



Arkansas 



Crossett Public Schools, Crossett 
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Harrison Public Schools, Harrison 
Rogers Public Schools, Rogers 
Russellville Public Schools, Russellville 
Texarkana Public Schools, Texarkana 



Mi nnesota 



Anoka -Hennepin School District #11, Anoka 
Blooming Prairie Public Schools, Blooming Prairie 
Centennial School District #12, Circle Pines 
Hutchinson Public Schools, Hutchinson 
Staples Area Vocationa 1 -Technical School, Staples 

Mi s si ssippi 

Amory School District, Amory 

Bolivar County Consolidated Schools, Cleveland 
Leflore County Schools, Greenwood 
Mississippi Gulf Coast Junior College, Gautier 
Pascagoula City Schools, Pascagoula 



Nevada 



Churchill County School District, Fallon 
Elko County School District, Elko 
Ormsby County School District, Carson City 
Washoe County School District, Reno 
White Pine County School District, Ely 

Replication of the System in Each State 

Activities expected of state leaders were listed under the following 
categories: selection of cooperating schools; orientation of local ad- 

ministrators and project leadership teams; completion of an agreement with 
each school; provision of leadership through a workshop; provision of 
monitoring and consultant services; conducting activities for reviewing 
progress of each school, including pericdic group meetings; preparing 
quarterly and final state reports; and evaluating local projects, and the 
state project. 

Minimum activities expected of all cooperating schools in each state 
included: a working committee of the faculty; a citizens 1 advisory com- 

mittee; a follow-up Survey of former students; and a statement of philos- 
ophy and objectives of the local or area program of vocational and tech- 
nical education. States were encouraged to add to these activities others 
particularly concerned with studying needs and local program planning, as 
well 'as any innovative vays of providing for vocational education. Each 
local school was required to prepare and submit progress reports and a 
final report on its project. 

In all four states a workshop was conducted to train local leader 
ship teams, followed by two project review meetings and a final project 
review meeting. Nevada conducted one extra project review meeting. The 
chief investigator assisted in all of these workshops, and in 11 of the 13 
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project review meetings. Consultants from outside the state were used in 
the workshops and in five of the project review meetings. Monitor visits, 
incluilng consultant services, were provided by state leaders to all 20 
systems . 

Each local leadership team was instructed to prepare * plan for the 
local program evaluation effort. Each also was party to an agreement 
with the state institution or agency. This agreement included: a 

descLiption of the project; the responsibilities of the state institu- 
tion/agency and/or state project leaders; the responsibilities of 
the cooperating school; financial considerations; and l^cal school per- 
sonnel responsible. 

All state and local leaders conducted dissemination activities at 
both the state and local levels. These included use of mass media of 
communication, as well as person-to-person communication. 

Procedure for Project and System Evaluation 

Evaluation of local projects included several approaches. The de- 
sign developed by the Multi-State Project leader was employed and reported 
to the state leaders. Depth interviews were conducted by state project 
leaders and summarized near the end of the second year of each state pro- 
ject. The checklist of faculty and citizens’ reactions to the activities 
in which they had engaged also was developed and recommended. 

The project review meetings were oriented to local project evaluation, 
as were also the final meetings. The local leader teams’ final project 
reports were submitted at the final meetings in each state. A consultant 
not connected with a local or state project assisted the state leader 
and the Multi-State leader in the final local leaders' meeting in each 
state to assist in auditing the results of the local projects in the light 
of objectives orginally set. The evidences bearing on the criterion 
questions set up for Objective No. 1 were planned to be summarized and 
analyzed at the final meeting of state project leaders. 

Work with State Project Leaders 

On March 18-20, 1968, a three-day workshop for the four state leaders 
was held at Michigan State University (See Appendix B) . Six consultants 
from Michigan and one from outside Michigan contributed to the program. 
Considerable time was used for individual conferences and group planning. 
Tentative dates for the state leadership training workshops were set. The 
manual on local program evaluation^ was made available in quantity, as 
well as several other instructional materials. Each state leadei pre- 
pared statements of objectives and a plan for conducting the project in 
hi? state. An agreement between MSU and each cooperating state agency 



1 . Evaluation of Local Vocational Education Progra ms- -Second Editio n , 
Department of Secondary Education and Curr iculnm. College of Education, 
Michigan State University, East Lansing, Michigan 1968, 83 pp. + vl. 
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or university was developed and placec into effect. (See Appendix C) 

Each agreement specified the responsibilities of the chief investigator 
and the state project leaders. Budgets were prepared and approved for 
each state project. 

The state leaders were brought together on three other occasions to 
share experiences, to receive instructions, to do cooperative planning, 
and to evaluate progress. These meetings were held in December, 1568, 

June, 1969, and January, 1970. (See Appendices D, E, & F) 

Periodic visits’ to cooperating states were made by the chief inves- 
tigator. One was made by the assistant on the project. Most of these 
visits were timed to coincide with a state workshop or project review 
meeting. Consultant service also was provided. At least one of the 
visits to each state included a stop at a c< operating local or area 
school system. Thus, six such visits were made. As needs arose, mater- 
ials for local leadership team assistance were made available to the states. 

Dissemination of materials to state leaders by the chief investi- 
gator was accomplished, in part, by sending numerous materials emanating 
from one or more of the state projects, but also materials relating in 
general to evaluation of vocational education, such as reports of evalua- 
tion studies and other similar documents. The chief investigator took 
ever> opportunity available to report to professional meetings, graduate 
classes, and other groups, that were meeting within the cooperating 
states . 



Evaluation of the Multi-State Project 

The evaluation of state projects was accomplished, using the design 
that previously has been mentioned. The chief investigator's judgments, 
combined judgments of state leaders, and evaluation by an outsicTe con- 
sultant all were used to make the final evaluation of state projects and 
of the system of evaluation, and of local leader training. A final meet- 
ing of state project leaders was held to conduct the evaluation of ths 
attainment of the objectives Nos r . 1 and 2 of the Multi-State Project , 

Dissemination of the Mul ti -Stat e Project 

The brochure describing the Multi-State Project was prepared for 
use in general dissemination as well as in dissemination w* thin each 
of the cooperating ttates. (See Appendix M.) This, and numerous pre- 
sentations made at regional and national meetings were used, in part, to 
prepare the way for rture widespread dissemination during the third year 
of the project . 



Regional Workshops 



The major dissemination activity during this third year was the con- 
ducting of three regional workshops. These were established and organ- 
ized to meeL the needs of two typos of persons. The first of these were 
those educator.. In positions of state leadership who would be interested 
in providing training and assistance to local leaders in dir »cting local 
program evaluations, in their schools or who had been assigned responsi- 
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bill ties in evaluation. Such individuals were based in a university, in 
a state department of education or in a research coordinating unit. The 
second were local or area school leaders in vocational education. It was 
also desired to obtain in ter -communicat ion between both types of parti- 
cipants for their mutual benefit. 

Closely related to this dissemination activity was the revision of 
the manual on local program evaluation previously referred to. This 
third edition was produced to be made available to the participants of 
these workshops and to those in cooperating states as an aid ^n further 
dissemination in those states and in local leader 'raining activities. 

It could be said that this manual represented the mode for training 
participants in the workshops for their respective roles in local pro- 
gr 3 m evaluat ion . 

The brochures announcing these workshops were given wide dissemination 
to all state directors of vocational education, to state RCU directors, 
and to most head vocational teacher educators. (See Appendix L.) Sites 
for the workshops were Michigan State University, University of Nevada, 
and Memphis, Tennessee. Additional visibility was given through the 
AV Journal , through contact with the National Association of State 
Directors, and head teacher educators in the several vocational fields. 

An effort was made to kaep numbers down to the optimum for effective 
workshops. A maximum of 50 participants was set for each workshop. The 
general purposes were leadership development and dissemination of the 
results of the project, 

Each workshop was planned well in advance of the date scheduled for 
it. This planning was done during two days spent at the site of each 
workshop by the chief investigator and two state leaders. One of these, 
in each workshop, was designated as co-director of the workshop. 

Inputs at each workshop included presentations by the chief investi- 
gator, by two state project leaders, one of whom also served as co-direc- 
tor of the workshop, by a local leadership team ar.d by other persons in 
state or local leadership positions from the host, state or cooperating 
state. Other contributions came from state or local leaders serving as 
discussion leaders, as resource persons and/or as reader and consultant 
for participants* plans Outcomes expected were development of plans 
for state projects of local/area leadership training in directing their 
own evaluation, and plans similarly developed by local/area leader parti- 
cipants for their own local situations. Materials used included the 
manual, 3rd edition, local and state leader pl^ii forms, (See Appendix G) 
suggestions for group work sessions, copies of certain presentations, 
list of definitions of terras used, selected references on display, and 
related materials. In addition, a video-tape recording was shown in two 
of the workshops. The workshops were evaluated by the participants, using 
instruments developed fur that purpose. One form was developed for daily 
evaluation and another one for final workshop evaluation. (See Appendices 
H & I) 
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IV RESULTS AND FINDINGS 



A. Within Each State 

Analysis o^f Informat ton About Schools In Cooperatin g States 

Criteria used by state project leaders in selecting schools to 
conduct local evaluation projects in the four cooperating states were 
given in the preceding section. The application of these criteria 
resulted in schools and districts that were similar in many respects, 
yet which showed considerable diversity. 

Administrative support was promised in all 20 schools. Various 
geographical localtions were represented within each state by the 
schools selected. There was a certain degree of homogeneity present 
since 'rhe very small, and the large or complex metropolitan area type 
of school system were not included. The variations in size of program 
are apparent as this characteristic is set forth in Table 1. 

TABLE 1 

SIZE OF SCHOOL SYSTEMS IN COOPERATING STATES 



ARKANSAS 


School System 


Enrollment 




Cresset t 


3,132 


1-12 




Harrison 


2,155 


1-12 




Rogers 


2,934 


1-12 




Russellville 


3,006 


1-12 




Texarkana 


6,759 


1-12 




- 






MINNESOTA 


Anoka 


28,000 


K-l 2 




Blooming Prairie 


1,269 


K-12 




Circle Pines 


3,500 


K-l 2 




Hutchinson 


2,5'<4 


K-12 




Staples 


405 


Post H.S. 


MISSISSIPPI 


Amory 


2,150 


1-12 




Bolivar Co. IV 


5,058 


1-12 




Leflore Cj. 


6,520 


1-12 




Miss. Gulf Coast J.C. 


604FT 

3,957?T 


Post H.S. 




Pascagoula 


7,113 


1-12 


NEVADA 


Carson City (Formerly Ormsby Co.) 


3,957 


K-12 




Churchill Co. 


2,726 


K-12 




Elko Co. 


3,822 


K-12 




White Tine Co. 


2,695 


K-12 




Washoe Co. 


27,573 


K-12 



The size of the participating schools varied within states as well 
as among »^he .ooperating states. In one stite, in which all schools 
were organifctd grades 1-12* the f*ize varied from 2,155 to 6,759. In 
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another sta:e in which one school was a county unit system, one a con- 
solidated school and two separate districts, the variation was from 2,150 
to 7,113 grades 1-12, with the fifth school being a junior college 
enrolling 602, but in addition 3,000 post-high school students on a 
part-time basis. In the third state the variation was from 1,269 to 
28,000, K-12, for four systems and one system, a post-high school area 
school, enrolling 405 full time. In the state, in which all districts 
are county units, two had only one high school each and one had four 
high schools. The K-12 enrollment varied from 2,695 to 27,573, The 
interpretation of findings, and the conclusions to be drawn can not be 
categorized as necessarily general izabl e to schools smaller than 1,000 
K-12 nor larger than 28,000. 

The fields reported as asking represented in the vocational or 
technical education programs in the 20 institutions were similar in 
some respects, but also varied, as well, as shown in Table 2. 

TABLE 2 



NUMBER OF COOPERATING SCHOOLS IN EACH STATE 
HAVING PROGRAMS IN GIVEN FIELDS 



Field 



Ark . Mint , 



Agriculture 4 

Business or Office 5 

Distributive* 4 

Health Occupations 
Home Economics 5 

Industrial** 4 

Technical 



4 

5 



5 

4 



Other : 

Cooperative Education*** 
Occupational 
Or ientat ion 
Misc . 



Miss . Nev. 



3 

3 
2 
1 

4 

5 

1 



2 

5 

2 

2 

5 

5 

4 



2 



2 



1 



~otal 

13 

18 

8 

3 

19 

18 

5 



4 

2 

1 



May include cooperative education 

*May include both industrial arts and vocational industrial education 
fiay , or may not have training in more than one field. 



For example, home economics was offered in all but one. Business or 
office education was found in all but two, as was also true for indus- 
trial education. In both these program types, however, there were 
variations as to courses offered within them. Agriculture, also, was 
very common, being offered in 13 of the 20 schools. Distributive 
education programs, on the other hand, were in only eight, although the 
four schools reporting cooperative education programs might have 

3 
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included distributive occupations on a cooperative basis only. There 
were only three systems offering training in health occupation and 
five with technical programs. It would appear that all schools did have 
a need to study existent programs, the manpower needs, and the student 
needs and interests. 



The objectives of each local or area school evaluation project con- 
formed to those of the related state project and of the Multi-State pro- 
ject. Additional objectives, however, were stated by several schools in 
one state. In other states there m£.y have been additional local project 
objectives or purposes, even though they were not recorded. 

Results Within The States 

The selection, planning, and carrying out of activities in the 
cooperating school systems resulted in many activities comon to all, 
but also in some activities of a special nature, or unique to a local 
project. In one state all schools completed all the minimum required 
activities. In another state four of the five did so. In the third 
state all schools conducted all required minimum activities, but one 
school had not completed all of them at the time of the state final 
report. In the fourth state all minimum activities were planned. Three 
of these school systems completed all of these. One school did not 
organize an advisory committee and one did not prepare a statement of 
philosophy and general objectives of the local program. In effect, 
then, 18 of the 20 cooperating schools did complete the minimum activities 
for the project. Additional activities were undertaken in one or more 
states, and by one or more school leadership teams. Twelve schools in 
the four states tried out the Ohio Vocational Inter t Survey. This 
activity, as well as other special activities, will b*» presented in more 
detail at a later point in this report. 

Although the statement of philosophy and objective ? of vocational 
education in the local institution was an activity of nearly all cooper- 
ating schools, no comparisons of the results of this endeavor were made, 
nor was any summation of this activity attempted. It is known that 
several developed statements for each occupational program area or 
field. Most prepared a statement of philosophy of vocational education. 
Few, however, explicitly stated over-all objectives of vocational 
education for the school systems. Following are examples of statements 
of general or over-all objectives made by three schools in two states. 

Sample Qblec t ives From Three Schools 

-To provide a program that .reflects changes in occupations and 
employment . 

-To prepare employment-bound youth for the world of work 




-To instruct the student in employer -employee relations, and to 
provide training in personality development. 

-To help the individual master the basic skills of a vocation for 
which he has aptitude and in which job opportunities exist 
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••To develop desirable attitudes for successful employment 

-To provide basic knowledge in abroad number of Vocations 

The present study did not attempt to identify the reasons why 
certain activities were not conducted. There is no full explanation, 
therefore, for failure to include general objectives in the local pro- 
ject final reports, or to prepare well worded statements for reporting. 

It may be that administrators and teachers had never done this before. 

It is possible that they had not received adequate training for this 
activity. It may be that members of the faculty were primarily con- 
cerned with objectives for their own specialties; or possibly this was 
not stressed sufficiently by the chief investigator o** the state leaders, 
although the investigator did present a theoretical framework for de- 
riving such objectives. Schools in the Michigan Evaluation Systems 
project did not report the process through which over-all objectives 
were stated. In the final report of the project it was stated that 
'’Some schools paid a minimum of attention to ob jec tives . . . .This may be 
the most serious weakness of this pro jec t , "(5) It appears that an 
inadequate model was provided for this activity by the Michigan project. 

Other Activities Conducted 

Nine schools in three states used the Vocational Education Infor- 
mation Inventory, developed during the Michigan Evaluation Systems 
Project as an aid to local staffs in de\ 1 oping and stating a philosophy 
of vocational education and general objectives of local programs. Only 
one used the inventory to compare staff scores at the beginning and end 
of the project as one means of assessing the outcomes of the project, 
although this had been suggested to all state leaders as an optional 
activity. In that state the analysis was not completed due to failure 
to identify respondents in the administration of the inventory at the 
year's end. No ^orms had been established for this type of use, although 
juj / ratings of the items in the instrument had been obtained by the 
researcher and had been made available. No positive results are reported 
on this activity. 

Conducting employer interviews, or staff visits to industries and 
businesses was another activity, added in this case by all schools in 
one state and by three schools in two other states. This activity, as 
well as some others will be discussed in a later section of this report. 

Local Staff Involvement 

Since staff involvement was a required activity, and was considered 
basic to the success of a locally directed evaluation endeavor, it is 
pertinent to examine the extent and nature of it. All 20 local leader- 
ship teams organized faculty committees called "steering comrai t t ees" , 
"study comroi trees", or otherwise designated. As presented in Table 3, 
the range in number of faculty members involved, and the average for 
each state show marked similarity. Variations were greater within states 
than the averages among the states. These committees all included voca- 
tional teachers and most also included guidance counselors and other 
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TABLE 3 



STAFF INVOLVEMENT IN LOCAL PROGRAM EVALUATION ACTIVITIES 
IN THE TWENTY SCHOOLS OF THE PROJECT 



INVOLVEMENT 






STATES 






Range in number of faculty 
directly involved in each school 


1 


2 


3 


4 


Total 


including administrators. 


11-19 


* 


11-20 


11-21 


-- 


Total number involved 


71 


72 


76 


72 


291 


Average number involved per 


school 


14 


14 


15 


14 


14 


Average number staff committee 


meetings held per school 
*Range not reported 


3 


4 


4 


7 





teachers in the system. Administrators were not universally included. 
Some served as one of the two on the leadership team, while others were 
on the local committee. 

The average number of meetings held per school was similar among 
the states, with one ex.ception. This exception is a school in the 
fourth state in the tabl^. This school held 20 meetings. The large 
number of meetings of that particular school is accounted for, in part, 
by the fact that it did not use citizens 1 advisory committees, during 
the year of the project, although they had been used in previous years. 
The local leaders reported the belief that the faculty committee might 
have partially filled the intended role of a citizens r committee in some 
of its meetings. 

The meetings of faculty committees quite generally related to 
certain similar functions. These included orientation of and reporting 
to the local staff regarding the project, planning activities to be 
conducted, developing statements of philosophy and objectives for the 
vocational programs, describing and developing the school r s vocational 
education programs, reviewing follow-up and other survey forms, nomin- 
ating potential advisory committee members, providing communications 
regarding the project and processing and analyzing data. In one state 
two faculty committees met jointly with citizens r committees. One of 
these Joint groups developed a set of Job competencies. 

State leaders reported that much of the work of the project was 
done by individual teachers and saroll sub-committees. In mt ny schools, 
particularly the larger ones, sub-commit tees were formed according to 
program struct re such as agriculture, business, home econouics, etc. 
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The extent to which time was provided, or compensated for to work on the 
evaluation appeared to be related to involvement. In each state a few 
persons enrolled in a university for credit in a special problem or 
independent study. In one state this was done by a large number for 
"state certification credit. 11 In this state 27 individuals in the five 
schools enrolled with a university for credit in an individual study. 
This arrangement appeared to the state leader to bf at least partially 
effective in obtaining faculty activity in the local evaluation effort. 



Citizen Involvement 



The involvement of citizens in local program evaluation included 
staff interviews of employers, both in regard to curriculum and to stu- 
dent follow up. This activity was reported in a previous section. The 
major involvement, however, was represented by organizing and using 
citizens* advisory committees in the evaluation effort. 

The extent of use of citizens* advisory committees for evaluation 
in the four cooperating states is presented in Table 4. The explanation, 
in the case of the second state, of two school systems not using such 

TABLE 4 

EXTENT OF USE OF CITIZENS' ADVISORY COMMITTEES FOR EVALUATION 
OF LOCAL PROGRAMS OF VOCATIONAL EDUCATION 



EXTENT OF USE STATES 





1 


2 


3 


4 


Number of schools organizing and 
using committees 


5 


3 


5 


4 


Range in number of committee 
members per school 


15-45 


11-42 


20-29 


30-54 


Average number of committee 
members per school 


30 


18 


24 


38 



correnittees is that one schcol was delayed in the whole project by 
faculty negotiations and a change in leadership, and that the other was 
faced with special characteristics, being a pos t -secondary area school. 
Over 90 per cent of the school's enrollees within the latter school 
came from outside the district in which the school was administered, 
actually from all parts of the state; and placement of graduates was 
mostly outside the immediate community. Curricular committees drawn 
from industries relevant to that institution's pregrams had been used 
previously and were used during the project year. A revised plan for 
further use of a citizens' advisory committee is one outcome of the 
project. The one school in the fourth state not using a citizens' 
advisory committee Is /ro one previously reported as having held a very 
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large number of faculty committee meetings, and with these committees 
serving as the school's "advisory committees." 

The membership ranges and average number in a school in the states 
also are given in Table 4. It can be noted that there was a variation 
in the number of citizens serving on such committees in e*ch school. 

The record of those schools having general vocational education advisory 
committees, as well as departmental or craft committees was kept in 
some states . 

One state had an average of 5-11 committees per school, with a 
total of 36 different occupational committees, and three schools had 
general committees in addition to occupational field committees. In 
another state 53 citizens were involved as members of advisory 
committees in the three schools using such committees. The size varied 
from 5-11 . Two of these schools had faculty committees meeting jointly 
with citizens' advisory committees, with reportedly good results. In 
the third state three schools used already existing craft, or department- 
al advisory committees. Two schools organized new committees for 
evaluation which were reported by the leader in th£U state to have been 
more effective than the three that decided to use committees that had 
been previously established. In the fourth state four schools involved 
a total of 144 citizens in 18 advisory committees, or -n average of 
eight per committee. 

It is evident from Table 5 that the states did not follow a uniform 
practice with respect to choice of activities by cooperating schools. 

TABLE 5 

EVALUATION ACTIVITIES WITH WHICH LOCAL/AREA 
CITIZENS* 1 ADVISORY COMMITTEES ASSISTED 



ACTIVIT Y 



Public relations and dissemination 
Interpreting data relating to vocational 
education programs 
Appraising facilities and equipment 
Reviewing or formulating recommendations 
Developing instruments 
Making curriculum studies 
Planning for the evaluation effort 
Identifying vocational education needs 
Planning or conducting occupational surveys 
Interviewing industrial personnel 
Review or tour of vocational facilities in 
several schools 

Formulating statement of philosophy and 
h jec t tves 

Reviewing instructional matevirls 
Promoting a bond issue 



o 




NUMBER OF STATES 
REPORTING 

3 

3 

3 

3 

2 

2 

1 

1 

1 

1 

1 

1 

1 

1 
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Whereas a given activity was reported as having been engaged in, in 
three states, this could have been an activity of all the schools in 

these states or in only a few of them. Likewise, an activity reported 

by only one state might have been engaged in by more than one school. 
Within a given school, likewise, it is likely that craft or departmental 
committees, as well as general advisory committees might have engaged 
in a given activity. Furthermore, there might be some activities actually 
engaged in but not reported to the state project leaders. 

Program s tudv in relation to manpower needs , s tudent needs , and interest s 

It has been indicated previously that study of current programs 
in light of manpower needs, perceived student needs, and student 
interests is an activity that was recommended by all state leaders. A 

variety of ways in which this might be done also was suggested by 

them, so that choices might be made by those involved in the work in 
the 20 institutions. 

In one state eight different sources of recorded data relating to 
manpower needs were made available. Four of the schools' staff commit- 
tees analyzed these, and reported summaries of them to their citizens' 
committees. In one school the citizens' advisory committee did this. 

Two of the schools sought information directly from employers through 
teacher interviews. One leadership team and staff committee conducted 
an occupational survey of the area served by the school. One school 
surveyed part-time employment engaged in by in-school students. 

In a second state the local leaders focused attention on a state 
study which was already well under way at the time the project was 
started. Information had been obtained and analyzed from 1856 male 
students in schools which included four of those schools involved in 
4-he project. Through this survey a comparison was made between student 
anticipated employment and actual employment of these students' fathers. 

In a third state three of the schools studied manpower information 
in relation to programs. In one of these the teachers were released from 
in-school assignments to interview employers, using a uniform interview 
form. The results were presented and discussed by the advisory committee. 
In another school the advisory committee assisted in developing a list 
of competencies perceived to be needed by students as they completed 
the various programs and sought employment. 

In a fourth state a systematic effort was made to utilize data 
from the State Division of Employment Security. Four schools cooperated 
in this. One school was able to use an in-depth, occupatior i\ skills 
survey taken by this agency. Another school cooperated with the 
manpower sub-committee of the Model Cities Project. One school conducted 
interviews to 73 business firms in the area. These employers were asked 
to project needs six months, one year, two years, and thiee year.- in 
the future. Teachers in another school visited 31 places of employment . 
Another school received responses to 82 firms to which requests had 
been made for information regarding employment opportunities available, 
educational requirements and projected employment needs. One school 
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surveyed part-time employment of the student body and discovered that 
44 per cent were working for wages during the school week. 

Study of students 1 interests, beyond what already was being done 
through the guidance services of the schools in cooperating states, was 
more nearly uniform in one respect. All states, with varying degrees 
of participation, tried out the Ohio Vocational Interest Survey(OVIS) 

(8), The four states were provided 3,000 booklets. One state obtained 
additional survey booklets, through special arrangements with the 
publisher, and actually administered the OVIS to 3,358 students in the 
8th, 9th, and 10th grades. 

The purpose of trying out this instrument was to ascertain its 
usefulness in providing supplementary information about students' 
interests which might prove useful in answering the following criterion 
questions . 

1. Are the vocational programs offered in the school system 
adequate to provide for the range cf vocational interests 
of students in the system? 

2. Are the occupational plans and interests of the students 
congruent with manpower needs of the community and/or 
area to be served? 

3. Are actual enrollments in vocational programs commens urate 
with interests? 

4. Are the students enrolled in the areas in which they have 
tested interests? 

♦ 

In the state in which 3,358 students were tested all five schools 
tried out the OVIS. The use was hampered by a delay in processing 
results and by lack, at that time, of an interpretation manual and 
national norms before the concluding of the project. The following 
excerpt from the state project leader's final report reveals possibilities 
in its use. 




“All five schools reported definite plans to use the 
data in counseling students on their career planning. Four 
schools reported using the information as one guide for 
suggesting needed curriculum changes. Actual changes were 
reported in one school as due at least in part to the 
interest data. Two other schools indicated curriculum 
changes were in the planning phase. 

“Two schools reported that a very high percentage of 
their students (757. and 85%) were interested in taking 
some vocational courses. One school reported finding a 
considerable discrepancy between the vocational choice 
of the student and his highest tested interest. The dis- 
crepancy was found to be larger in the case of ninth 
graders than it was with tenth graders. They planned 



-27- 



33 



to use the difference in students' stated occupational choices 
ard their tested interests as supporting evidence of the need 
for a formal course in occupational or ientarion . 11 

"One school indicated a definite desire to continue 
using the instrument with eighth grade students. They planned 
to use the Information to help them identify students who 
are interested in vocational subjects. A second school 
indicated further testing will be done at a later date in 
order to identify other specific areas of interest." 

In another state the instrument was used in just one school system. 
It was administered to students in the 10th grade in two high schools of 
the county district. Outcomes include the following: 

Comparative interests of students in two schools, previously 
unavailable 

Revelation of gaps in program offerings as indicated by surveyed 
inter es ts 

Evidence of need for reorganization of one (traditional) program 
Evidence of need for occupational orientation 

In another state the survey was used in the four secondary school 
systems in the project. In this state it was also desired to compare 
test results with those from other instruments being used in the four 
systems. Reportedly, one school used the results extensively in planning 
vocational course expansion for the program. In another system the 
results appeared to support the continuance of programs then extant. 

In the fourth state four schools used the OVIS with one or more 
classes. One school found agreement between measured interests in total, 
and interests claimed by pre -graduating seniors of tbe previous year. 

This school was able to identify needs for new vocational courses, 
based on students' measured interests. The cooperating schools indicated 
the values of the instrument for information helpful in career planning 
below the 11th grade. 

The chief investigator and his assistant provided guidelines for 
analyzing OVIS data, based on experiences of the state and local leaders, 
and study of profile, charts available. These guidelines were further 
revised in light of experiences in the projects, and are included in 
the Manual, Third Edition, previously referred to. 

Other activities contributing to evaluation of curricular programs 
and to program planning were conducted, but not by all schools nor 
in all states. Some of these are discussed at this point under the 
designation of special activities. 

Special local activities were engaged in by some of the schools 
which hopefully would contribute to the total effort of studying the 
existing programs. In two schools in which parent interest surveys were 
conducted the findings are relevant. In one of these, responses were 
obtained from 668 parents of children in the 7th, through 10th grades. 

In the other the parents of students in the 10th grade were surveyed. 

o 
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In both instances the parents* reactions to program offerings were 
obtained, and in one case their reactions to the proposed es t ablishmant 
of an area vocational education center was sought and obtained. 
Suggestions for improvement of program were obtained in both cases. 

One school had two special curriculum committees studying the need 
for occupational education at the elementary and junior-high school 
level. In another state a local leader conducted a student career 
aspiration study. This was not a part of the local project, however, 
the findings were considered in relation to other information and in 
making recommendations for improvement of programs. In a different 
state a home economics teacher in an opinionaire suivey of all girls 
in the high school, 9 through 12, received expressed vocational 
interests, career plans and reactions to many aspects of the existing 
program. This survey was also given to girls enrolled in home economics 
in two other schools. 

In one school in another state an occupational survey was conducted 
to determine occupational patterns, income, and job titles of students’ 
parents. Information yielded by this survey was utilized in planning 
and expansion of adult vocational educatioi> programs for the system. 

Follow-up Studies 

The activity of conducting follow up surveys was expected of schools 
in ell four states. In one state four schools conducted follow-up. In 
another, all five schools did so. In a third state the four secondary 
systems conducted surveys and the junior college system completed plans 
for the survey but had not conducted it by the close of the state 
project . 

In the fourth state, however, the State Research Coordinating Unit 
had almost concurrently completed a state-wide follow up of former 
students enrolled in vocational courses. Even though this did not 
include all former students in the schools, and in this sense :as 
incomplete, still all five schools had cooperated and had information 
at least In regard to former vocational enrollees. Likewise, the 
survey of in-school students previously referred to was also being con- 
ducted almost concurrently, and had provided substantial information 
for program planning. A number of other state-directed activities 
either obviated the need or competed with local leaders* or staff Lime 
so that no new mailed questionnaires were sent to former students. All 
live schools did, however, conduct follow up of former students through 
interviews with employers. A total of 180 employers were interviewed 
in the five districts served. Also one school conducted a follow up of 
former home economics students. Two other teachers in this school and 
four in a second school also conducted informal surveys using instruments 
developed by them. 

In the other three states 12 secondary systems, one pos t -secondary 
system and one junior college were involved in follow up. Of the 12 
systems conducting follow ups of former secondary students, five involved 
one graduating class each, four involved two classes, two involved three 



classes, and one follow up was confined to former students from four 
classes. In all these cases all but two schools surveyed all the former 
students, rather than just those who had taken vocational courses. The 
two cases were large systems in which random samples were drawn and 
followed up. In only one of these the follow up was limited to former 
enrollees in vocational courses. 

All but two of the classes selected for follow up had been out of 
school two or more years. One class followed up had graduated a year 
previous and the other had graduated in the survey year. Percentages of 
returns reported varied from 44 per cent to 89 per cent. 

The experiences of the 14 schools in conducting mailed surveys 
prompted the drawing of a number of inferences in regard to (a) value 
or usefulness, (b) follow-up procedures, and (c) possibly inhibiting 
conditions . 

The experiences of these schools seemed to support the following 
as the kinds of information most likely to be of value: 

-Adequacy of preparation for the job 

-Job satisfaction 

-Wages 

-Advancement 

-Time necessary to get job. 

-Job mobility and location of first employment 

-Whether further training had been taken 

-Desire for additional training, kind of training 

-Supervisors 1 rating 

Two procedures were demonstrated as having a positive effect on the 
values of follow up. One of these is the stating of objectives and 
criterion questions first before items are listed in a follow-up form, 
and the e x:\us ion of questions which # ould not contribute to testing of 
a stated objective. The second ~oc . .re or practices validated is 
that the more staff members are involved in the follow up, the more 
likely they are to use information from returns. This does not mean, 
however, that each should help tc word items to be included in an 
ins trument . 

The chief investigator and state leaders had stressed established 
techniques and procedures for follow up, both in workshops or project 
review meetings, and in consultant work with local or area school 
project leaders. Certain procedures included among these, as well as 
others, were demonstrated in the cooperating schools as being important 
and/or valid. These are as follows: 

-The instrument should be pilot tested before using. It 
may well be tried out with in-school students to eliminate 
ambiguous word, or phrases. 

Endorsement of a well known and respected person should 
appear in a cover letter 
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“The obtaining of addresses of former students and mailing 
should be the responsibility of just one person* 

-Telephone contact should be used, rather than a second 
follow-up letter to elicit delayed responses because it is 
more effective. 

-To obtain valid addresses, inquiry should be limited to 
more recent graduating classes, even though more useful 
data may be obtained from more experienced graduates. 

-Personal interview would appear to be the only practical 
way to follow up drop-outs 

-If the number of respondents is large, electronic data pro- 
cessing should be used. 

-Currently enrolled students may be used effectively when 
staff time is a critical factor and/or insufficient pro- 
fessional clerical help is available. 

-The months of December through March appear to be best for 
mailing follow-np instruments. 



The failure of a school to make a follow-up survey, or the making 
of a follow up thct dees not yield reliable, valid, and useful infor- 
mation may be due to lack of observance of some of these procedures . 

It may also be due, however, to the presence of certain inhibiting 
factors identified in the cooperating states. 

One of these inhibiting faclors is the occurrance of previous, or 
simultaneous state - directed surveys of a similar nature. The practice 
of limiting the follow up to former students enrolled in reimbursable 
vocational courses is another. In some cases these two inhibiting 
factors occurred together. A third inhibiting factor identified is that 
of a lack of clear-cut policies with respect to over-all program 
objectives and clientele. 

Many of the schools conducting follow-up surveys did so for the 
first time. This is also true of many of the local/area team members 
and cooperating staffs. There are, on the other hand, strong endorse- 
ments of this activity as a major one to include in a local program 
evaluation. There is also considerable evidence of the use of the 
findings, along with other information, for drawing inferences as to 
need for changes or additions to local programs to help make them more 
relevant or effective. 

Special Ac t ivi t ies 

Varied activities were engaged in by one or more schools, cs is 
set forth in Table 6. Eight schools In three states conducted employee 
interviews or staff visits to industries or businesses. Ono of these 
states, the one which a mail survey of former students was not con- 
ducted, had all Jive schools involved. A total of 180 employers, thus, 
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were interviewed. Reportedly, considerable information was obtained 
bearing on unspec talized competencies being sought by the employers. 

Six schools in three states conducted parent surveys. In the main, 
these were made to; (a) obtain parents' desires regarding program 
offerings for their youth, and/or (b) information about parents' occupa- 
tional statQs as a basis for planning adult vocational programs, but 
also with implications for secondary and post-secondary programs. 

The five schools in two States conducting surveys of in-school 
students did so for either of two reasons: (1) to get information on 

part-time employment held, or (2) to inventory students' vocational 
interests, career plans, parents' occupations, and evaluations of 
available offerings. 

The five schools cooperating with the R.C.U, state-wide follow-up 
system were all in one state. Because of the unique character of this 
activity the following description is taken from the state project 
leader's final report: 

"The objective of the try-out of the state R,C.U. System 
was to determine whether the computerized system developed 
for post-secondary students could also be used successfully 
With secondary students. 

"The system consists of three basic data collecting 
instruments, an entry personal data form, an exit infor- 
mation form, and a follow-up ins trument .. .From these in- 
put instruments, enrollment and follow-up suimaary reports 
are generated and sent to the cooperating schools. 

"Copies of the instruments were made available, and 
the operation of the system offered to the project leaders... 

All of the project leader s ,,, ,d ecided to participate. 

Several leaders said they viewed the system as a possible 
way of getting a follow-up procedure established on a 
continuous and computerized basis 

"The entry personal data form was administered to 
approximately 775 seniors in the fall of 1968. Four 
schools elected to have only their vocational students 
complete che form while the fifth school administered it 
to all seniors. By March of 1969, all the data had been 
key-punched and computer print-out reports giving names, 
addresses, and other basic information sent to the 
respective schools. 

"in May, most of the same students completed the 
exit information form. Again, the new data were key- 
punched, and up-dated reports prepared and distributed 
to the respective schools. 

"in November, these same students were sent a follow- 
up instrument using a computer to print-out the names and 
addresses which had been obtained from the forms the students 
completed as a senior ., .Plans called for at lest ': three 
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mailings to those who had not responded but finan- 
cial restrictions allowed for only one mailing. 

Some problems were also encountered with incomplete 
addresses and questions which were inappropr ia te 
for the secondary and especially for the non- 
vocational secondary student. 

"information obtained will be analyzed in 
the near future and reports sent to the respective 
schools. Delay in the follow-up analysis has 
been caused by the need to change original 
computer programs written for the university... 
computer, into a new format adaptable to the 
computer in the State Department of Education 
where the analysis will be made. 

"The try-out pointed up the need for rewording 
many of the questions. The Program Planning and 
Evaluation Section of the State Department of 
Education has assumed responsibility for charging 
and refining the system before extending its use 
to all secondary vocatioril students. The try- 
out of the original forms fias demonstrated that 
they need to undergo considerable modification 
and simplification if they are to be used success- 
fully with secondary students." 

Another special activity was a study of adult education in 
one school. The instructor of the adult farm management pro- 
gram was involved by the development of a follow-up instrument 
to be sent to families of former adult class students over a 
three-year period. An SO per cent return yielded information 
on perceived contribution made to the farm family, areas of 
instruction of most and least assistance, and opinion on 
priorities of various areas of subject matter. 

The study of elementary industrial arts also was a unique 
activity. The school system in which this was done had had a 
high drop-out rate in the elementary grades, and the community 
had had a high unemployment rate. A program was developed and 
planned through which potential drop-outs and other pupils 
would receive exploratory experiences and could develop some 
general competencies that might lead to employment in the 
world of work. 

Another special activity was referred to as a "teacher 
self evaluation." The focus of the instrument developed was 
primarily that of input, or process. This instrument consisted 
of 108 Items under the headings of training philosophy and 
objectives, adtrinis tr ation and facilities. The items were 
selected by the staff committee, with the assistance of the 
advisory committee and a consultant from the Stile Department 
of Education. Teachers checked presence or absence of the 



0 

ERIC 



( iU 



-y* - 



I 



characteristic or provision mentioned in each item. 

In one state a secondary school and a junior college 
identified needs for a school placement bureau as made manifest 
by staff and citizens' committees. The junior college estab- 
lished and filled a position of full-time director of the place- 
ment bureau before the completion of the project. Likewise, 
the secondary system established the combined position of 
counselor and placement director. 

In this state there was one school in which a number of 
other special activities were conducted or started. A simple 
vocational interest form was developed and used with 109 
juniors enrolled in the high school. The OVIS was used just 
with 200 pupils in the seventh and eighth grades. A pre-test 
and post-test was administered to 97 seniors in connection with 
use of an "occupational exploration kit." A survey of occupa- 
tions and income level of parents of all students was completed. 

All vocational teachers cooperated in preparing a "vocational 
handbook" for use in the guidance program. In addition, an 
"Aptitude for occupations test" was used with 250 juniors and 
seniors. Some of these activities might ordinarily be thought 
of as a regular part of a good school vocational guidance 
program. The project leader in this school was the guidance 
counselor. This fact may have been in part responsible for 
the inclusion of some of these activities in the project by 
the staff committee. 

Local Changes Made , or Recoimnenda tic ns Implemented 

It would be interesting to know in whac changes have taken 
place as a result of the evaluation projects in the 20 schools. 

Valid assessment of this sort would be difficult to make. 

Some of the reported changes might already have been tentatively 
decided upon or partially planned prior to the year of the 
local project. It covered only the academic ye r 1968-69 
plus, in some cases, a part of the 1968 summer and/or spring, 
and a few weeks of the summer and/or fall of 1969 prior to 
preparation of final local reports. Some reported changes were 
in terms of perceptions by leadership teams. While these 
are of interest they are without objective support, for the 
most part. 

There were five perceptions, nonetheless, that were 
reported to state leaders in considerable numbers and are 
worthy of note. These are--with number r epor t ing--improved 
statements of program philosophy and ob jec tlves (19) ; improved 
understanding ani a closer coordination among members of vocational 
staffs(14); improved understanding and support of vocational 
programs by administrators and school faculty (12); Increased 
citizen enthusiasm and support for vocational educa tion(8) ; and 
increased .ocational staff involvement with business and indus try . (8) . 
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Specific changes recorded, with number making changes, 
include the following: additional class sections in vocational 

courses(6); other vocational courses added(8); vocational or 
occupational orientation, exploratory programs or "exemplary 
programs" ins tailed (6) ; course revisions made (4) ; local 
director c vocational education position created'4); coopera- 
tive occupational education programs added or expanded(5); 
placement bureaus es t ablished (2) ; new equipment purchased (2) ; 
area schools approved, with financial support by voters (2); 
improvements in vocational guidance(2) additions to staff (2); 
revision o' the English programs(l); and miscellaneous (3) . 

In addition to the changes just mentioned every team made 
recommendations to its local decision makers in their final 
reports on their projects. These were not all necessarily 
included in the final project reports submitted to the state 
project leaders nor in the state leaders' final reports of 
state projects. Sufficient information is available, however, 
to indicate the major substantive content of the recommendations. 

The most common recommendation was that of adding courses 
to the curriculum, mentioned 15 times. These were almost 
exclusively specialized courses in vocational education. 
Rearrangement of, or additional space and/or facilities were 
mentioned three times. Adding a vocational teacher was 
recorded as a recommendation three times, as was also the 
development of, or acquiring instrument materials. 

Other recommendations reported relate to buildings, 
counseling and guidance, shared-time arrangements, and adding 
of a sub-administrator for vocational education. Three 
reconmendat ions of a different sort were recorded, relating 
to enrollment of students. In one case required enrollment 
in one vocational course for graduation was advocated. In 
another, it was recommended that all college-bound students 
be provided opportunity to prepare for part-time employemtn to 
help defray costs of education. In one instance permission 
Was recommended for students to enroll concurrently in courses 
in two different occupational fields when justified by students' 
career objectives. 

Follow up of local and area schools involved in the Multi- 
State Project subsequent to completion of their evaluation 
projects might have been desirable. It was not planned for, 
however, in the design of the project. It is reasonable to 
assume »;hat continued contacts with these schools have been 
made and will be maintained by state leaders in the four 
states. Evidence of the likelihood of continuance of local 
program evaluations was asked for by them in the final reports 
made by all 20 schools to state leaders, and through depth 
interviews conducted by these state leaders. 
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The resulting finding is that all 20 school systems 
planned to continue with locally directed evaluations The 
more corranon activities to be continued were reported to be 
citizens* advisory committees and follow-up surveys. In 
one state a locally directed evaluation in area post' 
secondary schools has been mandated for alternate years. 

This leads to the question of efficacy of the system 
from the view point of local or area project leaders a ui 
their administrators. Their judgments were in their 
final reports made by all cooperating schools to the state 
project leaders, and in the depth interviews conducted 
by these state leaders. 

In one state two schools reported that the system 
worked well. Three who reported that it worked "fairly 
well" indicated that time, extending through more then two 
years, would be needed, or that the fault was local rather 
than in the system. 

In another state the interviews revealed that the 
system worked well in four out of the five participating 
schools. This was also evident in the final reports of 
their projects. 

In a third state the following conclusions were drawn: 

"A comparison of specific objectives of the local 
projects with the accomplishments cited in the final 
report of each school indicated that all of these 
objectives were attained to a high degree. Further, 
the high level of satisfaction with the project re- 
ported by the local project leaders, administrators, 
vocational staff, and citizens provides additional 
support for this generalization. None of the personnel 
involved has in any way indicated disappointment with 
the overall project." 

In the remaining state the following statement was made 
by the slate leader on the basis of local reports and 
depth interviews. 

"Generally, local project staff, vocational and 
nonvocat tonal staff, administrators, and citizens 
agreed that local school objectives for this eval- 
uation project were achieved to a great degree. 

This concensus is supported by the lists of activities 
that were accomplished by each school system in the 
project. Further support for this generalization is 
based on the fact that enthusiasm for evaluation, and 
participation in project activities, actually increased 
as the project progressed. Also, favorable consents 
wete made in at least four of the schools that evaluation 
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efforts would be a continuous part of the vocational education 
program" 

S taf f and Citizen Reaction to Activities 

A check list of selected evaluation practices was developed 
and mad* available to state leaders to use in obtaining 
reactions from citizens and staff involved in the five coop- 
erating schools in each state. This was an optional state-leader 
activity and was in addition to depth interviews conducted 
by them. Because of these interviews and pressure of other 
activities the check list was used in just one state. The 
results of the reactions on the part of 85 citizens and 49 
local staff members are presented in Table 7. 

As may be noted in this table the citizens 1 reactions 
were positive ana strong to all activities except the last one* 
This is a practice in which most citizens were not involved. 

It is interesting to note also, however, that a considerable 
number of staff members also reported insufficient experience 
with this practice to evaluate it. Apparently the only prac- 
tice judged to be a poor one by as many as eight out of 49 
staff members was that of using a mailed questionnaire to 
employers regarding their satisfaction with former students 
now in their employ. This was also marked as a poor practice 
by moire citizens than any other one except no. 9. Both 
groups in that state apparently thought more highly of the 
interview approach. 



The Disserainatir n Phase 

The third phase of the project called for two major 
tasks. The first related to the final report of the project 
and the revised manual. The second, or dissemination task 
was to provide extensive communications in regard to the 
system being tried out and demonstrated in the project. 

This involved preparations, presentations and consultant 
service at state, regional, and national conferences, and 
in workshops . 
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Communication Through the Manual 



The revised manual is a major outcome of the project. 
While in one sense it is a revision of the manual, second 
edit ion! resulting from the Michigan Evaluation Systems 
Project^ it is in reality ? ^e-write based upon experiences 
in the four cooperating states. State project leaders 
provided inputs of information and examples of accomplish- 
ments in state and local projects. Instruments developed 
and used in the states were included in the appendix. The 
assistant on the project 1969-1970 wrote or assembled 
certain portions. The preliminary draft was read by all 
state leaders and a consultant, and suggestions were given 
in a meeting of these leaders and the consultant and 
project officer. 

The duplicated manual was disseminated in several 
ways. Copies were supplied to each state for further 
use there by state project personnel and local admini- 
strators. All participants, staff members, and 
resource persons in the three regional workshops were 
provided with copies. Leaders at the national level 
and others also received copies. 

Because of the anticipated wide interest in the manual 
approval Was requested and granted from the U. S. Office 
of Education for contract to have it published by a 
commercial printer and publisher^, Danville, Illinois. 

Comrounicat ion Through Conferences 

Throughout the three years of the project, communi- 
cations have been provided through various conferences, 
institutes, and workshops* Those in which presentations 
were made by the chief investigator, listed chronologically, 
are as follows: 



1. Harold M* Byram and Floyd McKinney, "Evaluation 
of Local Vocational Education Programs, Second Edition." 

E. Lansing, Michigan, College of Education, Michigan 
State University, March 1968, 83 pp & vi 

2. Harold M« Byrara, "Evaluation Systems for Local 
Programs of Vocat ional -Technical Education", Final Report 
Project No. 7-0211, Grant no. OEG-3-7-070211-1679, E. Lansing, 
College of Education, Michigan State University, October 

1968 129 & iii 

3. Harold M. Byram and Marvin Robertson, Locally 

Directed Evaluation of Local Vocational Education Programs, 
Third Edition . Danville, Illinois, The Interstate 
Printers and Publishers, December 1970, 97 pp. & vil 
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DATE 


NAME OF CONFERENCE. INSTITUTE OR No. 


OF PAR- 


LOCATION 


SEMINAR TiemNTS 




1968 


F eb . 
May 


Central States Seminar on Agri- 






cultural Education 

National Seminar on Evaluation, North 


120 


Chicago, 111. 




Carolina State University 


45 


Raleigh. N.C. 


July 


Central Regional Conference, 

Research in Agricultural Education 


45 


Lexington, Ky . 


0c t . 


Minnesota Department of Education 


20 


St. Paul, Mirm. 


Oct . 


National Conference on Evaluation 


165 


Atlantic City 


Dec . 


Research Department, American 
Vocational Association 


150 


Dallas, Texas 


1969 


Feb. 

Feb. 


Michigan Council, Local Administrators 
of Vocational Education and 
Practical Arts 

S. W. Michigan School Boards and 


70 


Lansing, Mi. 




Administrators 


100 


Lansing, Mi. 


June 


Faculty and Graduate Students in 
Vocational Education, University 
of Arkansas 


25 


Fayetteville 

Ark. 


Aug. 


National Institute on Improvement 
of Vocational Education Program 
Evaluation (U. of Arkansas) 


100 


Fayetteville 

Arkansas 


Aug. 


National Conference on Evaluation 
of Vocational -Technical Education 
Program Effectiveness, (U.C.L.A) 


30 


Los Angeles 


1970 


Feb . 


USOE Conference on Evaluation 


25 


Washington, D.C 


May 


Workshop on Evaluation of 
Vocational Education 


70 


Louisville, Ky . 


June 


Conference on Accreditation of 
Post-High School Institutions 


70 


Atlanta, T a. 


July 


Michigan Conference, Teachers of 
Vocational Agriculture 


150 


E. Lansing, Mi. 


Aug. 


USOE Conference on Evaluation 


25 


Washington D.C. 


Sept. 


Central Kentucky Workshop on 
Program Evaluation 


70 


Lexington, Ky . 


Dec . 


Agricultural Division, American 
Vocational Association 


75 


New Orleans, La, 


1971 


F eb . 


Central States Seminar on Agricultural 
Education, Program on Evaluation of 
Vocational Agriculture 


125 


Chicago, 111. 


Feb . 


Workshop on Local Program Evaluation 


35 


Kenai, Alaska 



o 



From the foregoing list it may be observed that eight presenta t ions 
at the state level to 540 educators, three presentations at the regional 
level to 290 educators, and nine presentations at the National level 
f-o 685 educators were- given, or a total cr 20 presentations to approximately, 
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1,515 educators, with only a few duplications. 

Presentations made by the chief investigator or his assistant to 
university graduate classes or workshops were supplementary to these con- 
tacts. Three were given at the University of Michigan and 11 at Michigan 
State University. An estimated 200 teachers, administrators, and other 
graduate students were involved in this way. 

Communication Through Publ ication and Audio Visual 

A brochure describing the project was printed and given wide distri- 
bution in several of the national meetings, and through the state project 
leaders at state and local meetings within the cooperating states. An 
article on evaluation in which the Multi*3tate Project is mentioned was 
written by the chief investigator and published in the Journal of 
Indus trial Teacher Education . The American Vocational Journal published 
an article announcing the regional workshops sponsored by the project. A 
news release was prepared and given wide distribution at the start, and 
another one near the end of the project. Local newspapers in Nevada also 
carried two stories relating to the projects. 

State project leaders added to these communications by two radio pre- 
sentations in Minnesota and one in Nevada. An article was published in 
the Min nesota Vocational Association Viewpoints and in the (Nevada) 
Vocational Re f 1 ec tor . A brochure was published describing the Arkansas 
project and given wide dissemination in that state and a national insti- 
tute on evaluation. Local leaders from three Arkansas schools also re- 
ported on their part in the project at a National institute. Dissemination 
also resulted from duplication of 100 copies of the Arkansas final report. 

A half-hour video tape was made of the presentation of the local 
evaluation project at Blooming Prairie, Minnesota as presented by the 
leadership team of that local school system. Copies of the tape have 
been obtained by Minnesota and Kentucky, and Alaska, and have been shown 
an unrecorded number of tiroes in these states. It has been shown twice 
to M.S.U. graduate classes and once each at the workshops at University 
of Nevada in August and Memphis in November, 1970. This video tape 
constitutes an illustration of the use of the evaluation system in one 
school. It has served as a motivation for other local educators. There 
are indications that further uses of this audio-visual aid will be made. 

Regional Workshops 

The major activity in the dissemination phase being the three work- 
shops, it is appropriate to deal with them in some detail. The purposes 
of the workshops were: 

1. To prepare state educational leaders to conduct programs for 
preparing local leaders in methods of directing or conducting 
local program evaluation. 

2. To prepare local or area educational leaders to direct and con- 
duct self-inJ tiated local program evaluations. 
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3. To dissemlna^ and review information about program evaluation 
gained in the Multi-State Evaluation Project. 

The applications for participation in these workshops totaled 184. 

Of these, 176 were accepted. There were 44 cancellations by these 
applicants, leaving a total of 132 participants in the three workshops 
divided 49, 44 and 39 respectively. All states were represented except 
Delaware, Colorado, Louisiana, Maine, Massachusetts, Rhode Island, and 
Virginia. State departments of education were represented by 34 partici- 
pants, 31 came from universities and colleges, seven from research 
coordinating units, 22 from local districts, 37 from area, or community 
college districts, and one was a state advisory council member. The 
numbers from each state varied from one to 15, with a median of three. 

If the 13 persons in state leadership positions and 40 in local leader- 
ship in the four cooperating states are added, a total of 185 educators 
have received training in locally directed evaluation of programs of 
vocational education through the Multi-State Project. These are in 
addition to 251 other local educators involved in the activities of the 
locally directed evaluations, or a grand total of 436 persons. 

Table 8 shows the distribution of the participants by states and 
area of professional employment or responsibility. The number of part- 
icipants from RCU f s actually exceeded the number indicated, but were 
counted under state or university categories if this was their major 
affiliation or base. The third workshop differed from the others in that 
a much smaller number of participants came from local schools as contrasted 
with area schools. In as much as the target groups of the workshops were 
persons in state leadership positions it is noteworthy that 34 states 
were represented by such persons. 

The analysis of information about participants shows several other 
things. One is that a desire for, or ability to participate in training 
in locally directed evaluation was shared by personnel in nearly all 
states, the exceptions being mainly in the Northeast. Another is that 
this is just as true of university-based personnel as of state departuunt 
personnel, even though the latter more frequently mentioned in their 
applications a commitment to, or mandate regarding evaluation. By far 
the greater majority of participants in the local/area category were 
adrai nis t ra tor s in area vocational schools, technical institutes, or 
community colleges. It is a fact that local districts greatly outnumber 
area districts or schools. A reason for their participation may well 
be that these institutions are under more presure to evaluate than those 
In local districts. On the other hand, no special effort had been made 
to recruit large numbers of local representatives. It is very clear, from 
information about participants and their situations, as described in 
their evaluation plans, that there are many differences among those 
local/area participants. There is little to suggest either the approp- 
riateness or practicality of setting uniform criteria for evaluation of 
local or area programs, nor any uniform specific requirements as to how 
such evaluations shall be conducted. There is evidence of corrcni tment to 
locai/area program evsluatior on the part of participants through their 
attendance in the workshops. Only three in the first workshop, and one in 
the second workshop die not attend the full four days. Two participants 
'^^e third workshop missed the last day due to unforeseen assignments 
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WORKSHOP PARTICIPANTS BY STATES, JOB LOCATIONS, AND WORKSHOP ATTENDED 
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by their employers received during the workshop. Thus, the attrition of 
participants was only 4.5 percent, and in terms of man - days, was only 
1*3 per cent. 

Staff members and others contributing to, or serving in a leadership 
role in the workshops were almost all connected with the Multi-State 
Project. Out of a total of 32 different workshop staff members, only 
six had not previously been connected with the project. Thus, in terras 
of staff input, these workshops constituted a valid dissemination 
medium. 

The manual on locally directed evaluation of local programs was used 
throughout all three workshops. It deals entirely with local or area 
program evaluation. Likewise, all of the presentations but two In the 
first two workshops dealt with applications of the system for evaluation 
to local/area schools or programs. One presentation dealt with the role 
of a state leader in training of local/area leaders in evaluation pro- 
cedures. Another dealt with relation of local evaluations to state and 
national evaluation. Both of these also appeared in the third workshop, 
but in addition there was presentation by a state leader not connected 
with the Multi-State Project, but who had worked with a previously project 
connected leader in his state to plan and conduct a state program of 
local leader training. Thus, there was less quantitative input relating 
to state leader's role than that relating to local leaders. But this 
was inevitable in workshops for dissemination of the Multi-State Project 
since the project dealt with local/area leadership. 

In all three workshops state leaders met together in separate groups 
for discussion of common concerns, In this way, in two meetings each 
in the first two workshops, and three times in the third one input was 
provided for state leaders in regard to their role and on planning and 
conducting state programs of local/area leader training. This was 
further augmented by individual conferences with members of the workshop 
staffs . 

Participants’ evaluations of each workshop were made during and at 
the end of each. In general, it is concluded that all three dissemina- 
tion workshops were very successful. The responses of participants on 
workshop evaluation instruments, as well as their oral comments indicate 
& high degree of satisfaction* Considering the extreme range in past 
experience, as well as diversity of interests and present job responsi- 
bilities of various participants, the workshops apparently met, and in 
many cases exceeded the expectations of participants. 

On request, many different individuals cited one or more aspects of 
the workshop attended as being a "highlight". Those responding to this 
request in the Central workshop were less numerous than from the Western 
and Southern group. Nearly every phase of each workshop was cited by at 
least one individual as being expecially helpful to that particular 
individual. Table 9 shows where agreement was greatest in highlights 
cited . 
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TABLE 9 



NUMBERS OF PARTICIPANTS WHO 
CITED SPECIFIC ASPECTS OF THE WORKSHOPS AS HIGHLIGHTS 

WORKSHOPS 



Highlights North Central 


Wes tern 


Southern 


Totals 


1 . 


Presentations 


4 


23 


28 


57 


2. 


Informal discussion 


18 


11 


12 


41 


3. 


Group work sessions 


0 


21 


17 


38 


4. 


Plan writing and 












sharing, under guid- 


1 


10 


8 


19 




ance 




5. * 


Process & method- 












ology 


1 


11 


4 


16 


6. 


Manual & other 












materials 


1 


7 


3 


13 


7. 


Interaction with 












consul tan t s 


0 


3 


7 


10 




As this table indicates 


, the presentations by 


staff members 


were 



considered to be highlights by more participants than was any other 
aspect of the workshop. The Nor th-Cent r al workshop participants were 
not as favorably impressed by the presentations as were those attending 
succeeding workshops. The fact that some of these criticisms were 
noticeable at this first workshop motivated staff members to taake im- 
proved presentations at the other two workshops. 

Very few of the other items on the last two workshop final evaluation 
instruments were included in the first workshop final evaluation instrument 
Therefore, it is difficult to tabulate or show directly comparable data 
for all three workshops on individual items other than the following: 

-There was nearly unanimous agreement among respondents from all three 
workshops that they had been sensitized to issues involved in planning 
and conducting locally directed vocational education program evaluations 

-The respondents of the first workshop were somewhat less satisfied than 
others that the Workshop had adequately identified the types of data 
and information required for evaluating programs. 

-The respondents in all workshops were in agreement, (with few excep- 
tions) that the manual should have been distributed and read prior 
to the workshops . 

-Even in the last two workshops, but especially in the first one, 
respondents believed more time should have been spent in discussions 
than acutally occurred. The changes in plans providing greater 
amounts of discussion at the two later workshops was an implementation 
of the evaluation responses at the first one. The improvement was 
^ ^eadily apparent in the responses at the later workshops, although 
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some participants reported the feeling that discussions were not 
adequately provided for in the schedule. 

-There was considerable uncertainty, and lack of agreement among 
respondents at all three workshops as to whether adequate time was 
provided for individual consultations with ?taff members during the 
writing of plan proposals. 

“Most respondents at all three workshops gave very favorable comments 
about group-work sessions. A very few individuals criticized one or 
more group leaders, but in general, most tended to communicate their 
feeling that the group-work sessions were very helpful. 

Although the second and third workshops appeared to be better 
received than the first one, all three were judged by respondents as 
being very successful in achieving the objectives established by the 
staff. In over-all evaluation of the three workshops, it was noted that 
experience gained in the planning and conducting of the first one resulted 
in several improvements in the later two, which were especially well 
received by the participants. 

Since the anticipated outcomes of the workshops included the writing 
of plans by participants in relation to their specific situation and 
its needs, emphasis was given to this in all three workshops. As a 
result of the first workshop, the plan outlines provided to participants 
for their use were revised for the other two workshops, and more time 
was provided for individual work on plans and for consultations with the 
plan readers and consultants. 

Every participant at each workshop was provided with instructional 
materials, was introduced to a workable process; was given stimulating 
and enthusiastic reports on typical evaluation efforts from cooperating 
schools; became involved in interaction with a variety of vocational 
education leaders from many states; was provided with guides to plan 
proposal development; was furnished opportunity for consultation with 
experienced personnel; and was encouraged to write a proposal for plans 
suited to their individual situations, for implementation upon return 
to their respective jobs. Almost every participant developed a proposal 
while at the workshop. Those whose plans were incomplete at the close of 
the workshop mailed them to the chief investigator. 

Pecause of limitations in the time schedule of activities of the pro- 
ject it had not been planned that a sy c tematlc follow up of participants 
would be made. They were encouraged to report voluntarily to the chief 
Investigator. Ten of those from the first workshop had done so prior to 
preparation of this report. Three state workshops and a region workshop 
have been reported as well as one local one, with several reporting 
extension of planning activities and contacts leading to local evaluation 
activities. Seven reported from the second workshop in regard to further 
planning toward local and state-led evaluation projects. Due to the date 
of holding the third workshop participants from just six states had 
reported by the time of preparation of this report. All these were state 
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leaders who were engaged in developing training programs for local leaders 
in evaluation. 

Dissemination by the Four Cooperating S ta tes 

Significant activities related to dissemination or continuation 
have taken place in several states. Of those which have been reported, 
four are most worthy of note. In Arkansas a credit course on planning 
and evaluation of vocational education has been added and taught at the 
university, and a special workshop will be conducted in summer, '71 
cooperatively with the State Department. In Minnesota a summer credit 
workshop on local program evaluation was taught, and two shorter ones 
were conducted for the State Department of Education. Another credit 
workshop is being planned for summer, 1971. 

It can be reported r 3s amatter of interest that in Kentucky a work- 
shop for state staff and one on evaluation of MDT programs have been 
conducted. In that state, also, a program of local leader training for 
Central Kentucky is being carried out with the starting workshop and 
project review meeting already having been conducted. A total of 38 
local leaders in 19 school systems are involved in that project. This is 
a state in which four state leaders had participated in the dissemination 
workshops . 



B» State Projects 

State Cbjectiv es 

All state project leaders prepared statements of objectives for the 
state project, and a plan for conducting the project and filling the role 
of state leader. All statements of objectives contained the three pre- 
viously stated for the Multi-»State Project. In some cases these were 
modified slightly to reflect the focus on local leadership. Additional 
objectives for each state also were presented. These additional state 
objectives follow: 

Arkansas 

- ,fc To assist in the development of leadership competencies that are 
needed for evaluating local programs of vocational and technical 
education . 11 

- ,l To create an awareness of the importance of involving both 
citizens and prof essi onals in the evaluation process. 11 

-"To foster the development of positive attitudes toward both pro- 
gram evaluation and vocational education. 11 

Mlnneso ta 



,fc To test the generallzabi 1 i ty of the evaluation system to post 
secondary and adult vocational programs. 




1fc To involve the State Department of Education, Division of Voca 
tional Education in the evaluation project as a prerequisite to 
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implementation of the system if it proved generalizable . M 

Mississippi 

•’'To aid four selected local public school systems and one junior 
college to develop and use a plan for evaluation of occupational 
education by utilizing local school per? innel, students, local 
community citizens, and by drawing on supportive state department 
and university consultant services. 

- rl To aid in the delineation of official and operative objectives 
and goals of occupational education programs in local schools, 
and relate these to observed outcomes. 1 ' 

-**To aid in explication of roles of individuals and agencies and 
the interactions which will lead to the identification of per- 
manent leadership experitse and citizen participant potential. 11 

-'To identify and delineate specific areas of the school -community 
developmental process which lend themselves to a research frame- 
work. 11 



Nevada 



- 1fc To determine how local schools throughout the state could become 
involved and become an active part of the evaluation process for 
vocational -technical education." 

- ,l To find a way that local school administra tors could become an 
active partner in the evaluation of local programs and involve 
the people who really counted in the development of local p ograms." 

- ,l Xo find ways and means of including a maximum effort of local 
faculty of the schools, using citizens and others with emphasis 
on program outcomes of the total school. The total effort be 
directed to answer the following question: Is the optimum pro- 

portion of youth and adults being adequately prepared for work in 
line with employment opportunities and their own aspirations and 
needs?" 

State Project Plans 

The plans prepaved by each state leader included orientation, pro- 
vision of workshops, monitor-consultant visits, project review meetings 
and communication-dissemination activites. The preliminary drafts of 
these plans were developed and discussed at the workshop held for state 
leaders and, with appropriate revisions, the plans were put into 
immediate operation by them and their assistants. 

State Leader Activities 

All state le 6Cier s made five orientation visits to local schools and 
Q discussed the projec: with administrators and persons to he involved in 
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local leadership. In one state seven visits were made because two add- 
itional schools had considered participating. 



Visits to provide consultant service and to monitor the local pro- 
jects were made in greater numbers. In one state 15 were made, The 
distances involved made a larger number than this not feasible. In 
another state 38 such visits were conducted, in a third state 21 visits 
and in the fourth state 28 visits. Assistance was provided in all phases 
of the project. With fewer individual contacts available in the first 
one mentioned, nonetheless, mailed service letters were used. Additionally, 
since some local leaders and many teachers were enrolled either for 
university credit or state certification credit, individual help was 
provided through these sources. 



A workshop for local projec 
leader in each state in the spri 
shops were helo for three days, 
addition to the assistant leader 
service was utilized from a Midi 
from the state university in 14 
of education in two states, tota 



t leadership teams was conducted by the 
ng of 1968. In two states the work- 
and in the other two for two days . In 
s and the chief investigator, consultant 
igan local leader in five instances, 
instances, and from the State department 
ling four persons. 



In the three project review meetings, including the final report 
meetings, held in three states, and four in the fourth slate, Michigan 
local Leader consultants were used in four cases, and a Michigan Depart- 
ment of Education leader in one case. Additionally, university consult- 
ants were used in 24 instances, and 24 consultants or resource persons 
from state departments of education, all from the cooperating states. 

All but two project review meetings were held for two days each, the 
remaining two were held for just one day. Thus, approximately nine full 
days of group training, on the average, in each state, and an average 
of over 30 days of individual consultant service were provided. The 
extensive and intensive preparation for this service, and the reporting 
and other services to local leaders represent a proportionately large 
input. Progress reports were received from local leaders in these 
meetings. Problems were raised and dealt with. Evaluation procedures 
were discussed. Plans for preparing reports were presented or developed, 
and suggested format for final local reports was provided. A sample for- 
mat is given in Appendix J, Instructional materials from the s'ate 
leaders,, their assistants, or other consultants or resource persons were 
presented ant discussed. 

Dissemination and Further Application o f the System 

Dissemination activities within each state varied. A feature in 
Nevada w£s a final report banquet attended by about 60 people. These 
included administrators and faculty from the University of Nevada, 
including the State Research Coordinating Unit, and from the Nevada 
Department >f Education. Local newspaper publicity was provided in 
Reno, and over a Reno radio station. 



In Minnesota, invited guests at meetings provided dissemination. 
These incited university and State department personnel. In addition, 
** considerable use was maue, subsequent to the project, of local leaders 
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and consultants In meetings. The credit workshop on local program 
evaluation featured dissemination of information of Minnesota local 
project activities. 

large numbers of State department of education and university 
guests provided dissemination in Miss Iss Ippi . Additionally, the periodic 
reporting on the project in publications of the Mississippi Research 
Coordinating Unit provided visibility. 

In Arkansas an illustrated brochure dascribing the state project 
and local projects was printed and given wide circulation, both within 
the state and nationwide. At two project review meetings graduate 
students, as well as State department and University personnel, as 
guests of these meetings, became better informed. 

The degree to which the systematic approach has been adopted in a 
more widespread manner cannot be reported in its entirely because of 
lack of sufficient contact with state leaders during the third year of 
the project that might have provided such information. It is known 
however, that in Minnesota two workshop sessions were conducted for 28 
post-secondary vocational education schools. About 20 subsequently 
conducted their own evaluations, featuring advisory committees and 
follow-up surveys. In 1970, a credit workshop for 23 local leaders 
was held to prepare them for conducting evaluations in their own schools. 
This tvpe of workshop will be repeated for other local leaders in summer 
1971. 



In Arkansas additional administrators and teachers have been given 
instruction on local evaluation procedures in graduate classes. Also 
in Arkansas the State Department and the University are cooperatively 
sponsoring a leadership training workshop for leaders from 16 schools 
to deal with a comprehensive local program planning and evaluation 
effort. This would be conducted in summer, 1971, with follow-up activities 
for fall and winter similar to that in the present multi-state project. 

In both Arkansas and Mississippi certain local, or area systems are 
known to be planning to direct their own evaluations. In Mississippi 
one outgrowth has been the development of a follow-up system for use in 
both local evaluations and In statewide evaluations. 



All state project leaders prepared and submitted quarterly progress 
reports to the chief investigator. A final report on each state project 
was submitted in January, 1970 (6, 12, 15, 19). These state reports 
were useful in preparing the final report of the present project. 

C. Revision and Publication of the Manual on Evaluation 
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The second edition of the manual on locally directed evaluation of 
programs of vocational education has been previously referred to as one 
piece of instructional material used in the workshops for state leaders 
in March, 1968, in all the state workshops for local leaders and in the 
project review meetings held for them. One result of this usage was 
feed-back information regarding the parts that were useful or helpful, 
and types of additional information desired by those who used it. Beyond 
^ .his , new activities and procedures and instruments were coming to light 
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that could be considered for inclusion in a revised edition. Many re- 
quests for copies of the manual, second edition had been received and 
could not be filled. 

It was because of this experience that assistance of the state 
project leaders and the assistant to the chief investigator was solicited 
early in the project. At the June 3, 1969 meeting of the state project 
leaders, suggestions for revision were discussed. It was decided to 
place more emphasis on staff and citizen involvement; to prepare an over- 
view chapter; to revise the instrumentation information on follow-up, to 
add a section dealing with study of students' interests; and in other 
ways to more adequately reveal local evaluation practices actually 
being demonstrated. 

A preliminary draft of the revision was subsequently sent co the 
state leaders and discussed at their final meeting on January 14, 1970. 
General suggestions, as well as specific editorial and content suggestions 
were received. 

Steps were taken to obtain approval for a limited copyright contract 
with a publisher for the revised manual. This eventually was accomplished 
It has been published by the Interstate Printers and Publishers, inc., 
Danville, 111. under the title of LocallyJDir ected Evaluation of Local 
Vocational Education Programs ^ Th ird Edition - a Manual for Administrators 
Teachers j and Citizens, 1971. A copy accompanies this report. 

Because of the need for the use of this manual in the three dis- 
semination worktops starting in April, 1970, and the time required for 
contracting and publishing, a limited number of copies of the manual was 
run off for restricted use. These copies were loose-leaf and punched 
for insertion in notebook covers of workshop participants. 

D. Evaluation of the System 

In this section the three major objectives of the project will be 
re-examined in arriving at an evaluation of the system and of the pro- 
ject, Under major headings of "general izability - feasibility" and 
"e f f ec tivene: s 11 of local projects, and gener alizabili ty of state systems: 
(a) criterion questions are stated; (b) findings and facts are given and 
(c) conclusions or answers to each criterion question for objective No. 

1 . 



Generaligablli ty and Feasibility - local Programs 



1. a. Were the minimum activities attempted in each school? 

b. The records show that all the minimal activities t^ere accepted 

planned, or attempted in 18 of the 20 schools. These 18 re- 
present all five schools In one state and four out of five in 

the other three. 



c. The conclusion is that the activities designated as required 
or minimal were found acceptable in nearly all cooperating 
schools . 
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Were the activities planned or attempted carried to completion? 

In 18 of the 20 schools all of the minimal activities planned 
or attempted were carried to completion. Beyond these 
minimal activities, 12 schoola engaged in study of the use of 
OVIS in evaluation; nine tried out the VEII with staff; and 
eight conducted employer interviews or staff visits to 
industries. In addition, 19 other activities were engaged in 
by from one to six schools* 

The conclusion ia that the cooperating schools found the 
minimal activities feasible. Moreover, in addition to these 
many schools conducted activities recommended by a state 
leader, by the Multi-State leader, or by local leaders or 
consultanta and found them feasible. 

Did cooperating schools vary as to size? 

Reported size varied from 1,269, K-12 to 28,000, K-12. 

No extremely small systems and no large, metropolitan 
schools were included. Only two systems had pupil enroll- 
ments over 8,000. 

The conclusion is that small or medium size systems are 
better represented by project schools than larger systems. 

Did cooperating schools vary as to type of organization. 

The majority, 11 out of 20, were local school districts. 

Seven were county districts, and two were post-high school 
districts . 

Schools did vary as to type of organization. Findings, 
can be generalized to local districts, but not necessarily 
to county unit or area diatricts. Variations were such 
as to make possible aome variation in application of the 
evaluation system. 

D*d schools vary as to type of programs? 

The school systems were uniform in one respect, in that all 
but aeven had programs in three or more occupational fields. 
Variations were represented in these schools with respect 
to other occupational fields. 

Similarities as to general nature of program, by occupational 
fields, were more characteristic of the schools than were the 
differences. The program designations by fields are not 
considered to constitute a factor that would prevent or cur- 
tail applicability of the system. 

Did schools vary as to objectives? 



b. 


Most of the schools had not previously stated over-all objectives 
for vocational education. Most had had statements of objectives 
by occupational field, Local and state final reports generally 
did not include these statements, although they were reported to 
have been prepared. 


c . 


The conclusion is that variability as to objectives of vocation- 
al education in cooperating schools is not known or i s in doubt. 


5 . a . 


Were variations of activities conducted in cooperating schools 
related to size, type of organization, type of program or 
objectives ? 


b. 


There is no evidence that the variations in activities were 
associated with size of the systems. There were variations 
by type of organization and type of program. 


c . 


It is concluded that variations in evaluation activities 
conducted in the several schools were generally unrelated to 
differences in size, to type of organization, to type of pro- 
gram or to objectives. 


6. a, 


Was faculty involved to a high degree? 


b. 


All schools had faculty ccmmittees for evaluation consisting of 
,( vocat ional n teachers, guidance counselors, and others, 
totaling 291 educators in the four states. The average number 
of meetings of these committees was similar in each state. 

The functions and activities of these committees were similar. 
In addition to these committees, individual teachers and 
smaller sub-committees were involved. 


c. 


The conclusion is that the faculties in nearly all the cooper- 
ating schools were involved to a high degree in a wide range 
of evaluation activities in the project. 


7 . a. 


Were citizens highly involved in each school during the 
project ? 


b. 


Seventeen of the 20 schools used citizens' advisory committees 
ranged from 11 to 42 per school in one state to 30 to 54 in 
the state reporting larger committee memberships, and an 
average per school, per state ranging from 13 to 38. Committees 
included some general advisory committees, as well as craft, 
or departmental advisory committees. There was considerable 
uniformity as to performance by those committees of six 
functions, however, there were seven other functions reported 
In a minority of the states. 



8. Conclusion: 

It is concluded that the system for locally directed evaluation of 
programs of vocational education was tried out and demonstrated to be 
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generalizabl e and feasible in nearly all of the schools in the four 
states participating. Whether this system is generalizable in county 
unit systems seems likely, but was not established. It has not been 
established that the system is generalizable in area vocational schools 
or community colleges primarily because of the limited number of partici 
pating schools of this type. There is little evidence to indicate, 
however, that it would not be general izable . 

Effectiveness of Sys tem - Local Programs 



1 . a . 


Have changes in local programs been planned and/or made, 
either directly or indirectly as a result of the project? 


b. 


-Improved statements of program philosophy and objectives wcj 
reported by 19 of the 20 schools. 

-From two to six schools reported program curricular, staffin 
and other changes made, under 12 different categories. 


c . 


It is concluded that changes in local programs have been 
made and that others have been recommended in the great 
majority of the cooperating schools. 


2. a. 


Has a change in attitude toward, or understanding of vocatir 
al education taken place in the faculty of each school 
during the project. 


b. 


-Judgments by local leadership teams in 14 schools indicated 
a perception of improved understandings and closer coortlinai 
among vocational teachers, 

-Judgments by local leadership teams in 12 schools indicated 
a perception of improved understanding and support of vocal 
al education programs on the part of administrators and 
school faculties. 

-Although nine schools administered the Vocational Education 
Information Inventory at the beginning of the project, only 
one did so at the end of the project and the data produced 
were not identifiable by individuals. There are no objectiv 
data, therefore, to indicate that the project was effective 
in changing faculty understandings in regard to vocational 
education . 


c . 


It is concluded that, if the judgments of local leaders win* 
worked with faculties can be assumed to be valid, then the 
system was effective in changing attitudes and understanding 
about vocational education. 


3. a. 


Has a change in attitude toward, and/or an understanding of 
the vocational education program of the school by the citi.z< 
involved taken place during the project? 


b. 


Eight schools reported either or both increased citizen 
enthusiasm and support for vocational education and incrcin 
voctaonal education staff involvement with business and 
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c , 


There is insufficient evidence on this question to warrant 
a generalization. It should be noted, however, that the 
purposes of involvement of citizens were other than that of 
changing their attitudes toward vocational education. 


4 . a . 


Will locally-directed evaluations be conducted in future 
years on a regular or continuing basis? 


b. 


Final reports of local projects, as well as depth inter- 
views conducted by state leaders of local administrators 
and leadership teams reveal that all schools involved plan 
to continue with local program evaluations in the future. 


c * 


The conclusion is that local evaluations will be conducted 
in the future by the cooperating schools. 


5. a . 


Were local citizens 1 and educators’ reaction to local eval- 
uation practices favorable? 


b. 


-State leaders' reports relaying information from depth 
interviews and local project final reports show a great 
majority of the schools reporting favorable reactions to the 
system (to the evaluation practices or activities) 

-A report from a sample school of the results of responses of 
85 citizens and 49 local school staff members to the 
check list of 16 practices revealed that the great majority 
of staff meiabers designated all practices as "good ,r and 
that the great majority of citizens designated all but one 
a "good" practice. 


c . 


It is concluded that the citizen and staff reactions to the 
system or to the activities or practices were favorable. 



Generallz ability o f State Systems 



1. a. 


Were objectives of state projects similar? 


b. 


All lists of objectives included the three previously 
stated for the Multi-State Project. The two to four 
additional objectives submitted by each state related very 
closely to these three objectives. They focused on increased 
state involvement and awareness of the importance and methods 
of locally directed evaluation. 


c . 


It is concluded that objectives were quite similar. 


2. a. 


Were activities in conducting state projects similar? 


b. 


-State project plans followed practically t He same outline. 
There was no state that omitted from its plan an activity 
expected ot it. There were a few additional activities 
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planned in some states in line with additional objectives. 

“All states provided a workshop for local leaders at the 
beginning. 

-Quarterly reports to the chief investigator were made regular- 
ly by the state leaders. 

-Monitor visits were made in fairly uniform numbers in each 
state, with reports of these included as a part of quarterly 
reports . 

“State leaders kept a log, or other running record of their 
own activities. 



-Three project review meetings were conducted in all states. 

-All states mede use of consultants from Michigan experienced 
in directing local program evaluations. 

-Memoranda, newsletters and other appropriate means of comm- 
unications were used in all states. 



c. The conclusion is that activities of conducting state projects 
were similar, in fact quite uniform in kind and number. 

3. a. Were state project plans and activities adapted to differences 
among the states? 

b. Several differences were related to different situations and 
objectives in the states. 




-One state adapted the procedure to schools administered in 
county units. Activities also were modified due to limited 
leader time available and distances to cooperating schools. 

-Two states selected cooperating schools for the project 
because of interests in, and presence of a variety of situa- 
tions such as pos t -secondary schools, adult programs, and 
county unit systems. 

-Three states were aided by personnel and/or financial support 
from the Research Coordinating Units which made a difference 
in the scope or quality of conduct of some activities. 

-One state provided for considerable visibility or communications 
in line with objectives for that state. 

-The two states in which there was a change in project leader- 
ship conducted some activities differently than otherwise 
Would have been the case, but did not actually add to nor 
subtract activities from those previously planned. 
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c. The conclusion is that state project plans were adapted 
somewhat to differences among the states, and activities 
reflected to these differences, but were basically very 
s imi lar . 

Conclusion 



It is concluded that the system for providing state leadership to 
the local school systems in the cooperating states were similar to one 
another and to the cooperating states. 

Ob jec tive No . 2 ■ 

This objective was stated as n to discover and/or develop new or 
improved procedures in a state system for local evalua tions . 11 

a) C r iter Ion ques t ion : To what extent were such procedures dis- 

covered or developed? 

b) Evidence : 



-For the first time state projects were conducted with help 
from staff resources of the State Research Coordinating Units 
In three states and financial resources in two states. 

-A new practice was that of providing all cooperating schools 
with a suggested format for final local reports. 

-For the first time instructions and interpretation procedures 
were provided for use of OVIS in local program evaluation. 

-Two states provided for individually reviewing preliminary 
drafts of final reports at semifinal, local project review 
meetings . 

-In one state effective analysis of data in follow-up studies 
by all schools was facilitated by making arrangements for 
computer use at the university. In this state there was some 
coordination of this activity with the state-wide follow up. 

-One state successfully employed a person not previously 
connected with the project to conduct structured depth inter- 
views of administrators, local staff members, citizens and 
former vocational students. 

-One state successfully used a summer meeting of project leader 
teams to supplement project review meetings that were other- 
wise uniformly scheduled during the academic year. 

One state made effective and rather extensive use of state 
certification credit, and/or university credit, independent 
study projects to encourage individual teachers to partic ipate 
in the local project, and in seme cases to engage in unique and 
supplementary activities in evaluation. 
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“One state demonstrated that a local program evaluation can 
Include an adult education program 

-The many new, or improved procedures developed and/or tried out 
necessitated a revision of the manual on local program evaluation 

c) Conclusion 

All states developed, discovered or tried out four new or improved 
practices and two tried out five new or improved practices. Since these 
were in addition to those originally planned at the start of the entire 
project it is concluded that new or improved procedures were developed 
or discovered and tried out. 

Objective No . 3 . 

This objective was stated as r, to assist in de 1 elopment of state 
leadership competencies in evaluation of local programs of vocational 
and technical education. . « 11 

a) Criterion question : Did this project assist in the development 

of these state leadership competencies? 

b) Evidence : 

-Nine persons in state leadership positions and eight of their 
assistants were involved in activities that have developed such 
competencies . 

-Of these nine persons, six filled the role of leadership contri- 
buting to successful completion of the state projects, and two 
have qualified for similar positions of leadership and are 
currently conducting state evaluation .pro jects . 

-Of the eight assistants, all but one are now in state leader- 
ship positions. 

-Development of competencies have resulted in conducting credit 
workshops or courses on evaluation in two states, in conducting 
a national institute on evaluation in one state, and in consul- 
tant service by leaders in three states extended to other states 
or national groups. 

-Six persons in state leadership positions have effectively assis- 
ted with the conducting of the three regional workshops for 
State and local leaders on locally directed program evaluation 
for 132 educators from 43 states. Of these, 73 were in positions 
of state leadership in 37 states. 

c) Conclusion : 

This project has, indeed, contributed to the development of competen- 
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cies in evaluation of programs of vocational education. While there is 
insufficient evidence to show that the potential of the involved leaders 
was developed to the maximum, there is sufficiexit evidence that the 
potential values in many of the activities engaged in were rather uni- 
formly maximized. 

General Conclusion s 

1, The feasibility and general izability of a state procedure for 
assisting local district leadership to ose the evaluation 
system developed in Michigan has been demons crated . 

2, New or improved procedures in a state system or plan of assis- 
tance for local program evaluations have been found or developed. 

3, The project provided assistance in the development of state and 
local leadership competencies in evaluation of local programs 
of vocational education. The values of local involvement were 
realized . 
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IV. CONCLUSIONS AND RECOMMENDATIONS 
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Conclusions 

The system for locally directing an evaluation of a program of 
vocational education, and for preparing local peLSonnel for conducting 
such evaluations as developed and tried out in Michigan was further 
tried out and demonstrated in the states of Arkansas, Minnesota, Missis- 
sippi and Nevada. It was found to be replicable in these states and, 
with minor exceptions, in the schools involved in these states. The 
move specific conclusions on which the foregoing genera lizat ions are 
based follow. 



1. From the accomplishments reported in this project, it ap^ars 
that all states could meet the criteria used for selection of 
states, with the possible exception of criterion number three, 
relating to state administrative endorsement, and sufficient 
leader time and supporting resources. The representativeness 
of the states selected was not established. Little evidence 
was available, however, to indicate tha other sta is could 
not do what was done in the general form involved in the study. 
On the other hand, it is possible that some other states not 
only may have a stronger commitment to encouraging and a lis- 
ting with locally directed evaluations, but may have mor near- 
ly ample s*^ff and material resources with which t:o work. The 
need for resources of staff and financial support ha? been 
Indicated by this study. 

2. The replicability in local districts that are smaller or medium 
in size, has bet n established. There is little evidence, how- 
ever, to indicate that the system could not be successfully 
used in larger local systems In area secondary and post-secon- 
dary systems, and in community colleges. The participatory 
response to the pro j ect -sponsored regional workshops shown by 
37 of the 132 participants who were connected with such insti- 
tutions is indicative of interest in or expectation of the 
applicability of this system to their institutions. 



3. The replication at the state level was validated by equivaler es 
among these states of commitment to state leader time, and ot 
state advisory committees to the project, state project plan 
and project objectives, as well as to all major local leader- 
ship training activities. Special situations contributed to, 

or necessitated only minor changes in the states involved in 
their approached to local leader training. These situations 
are also reflected in new or improved practices which may 
have enhanced the general procedure. 

4, Replication of the system in local schools was characterized 
by general uniformity, in terms of objectives planned and 
carriec out. Additions were made in some schools which may 
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constitute improvements in the system. It was found that the 
system was sufficiently flexible but that in some instances a 
more structured format could well have been provided. 

The evidence in regard to attainment of all three major purposes or 
objectives of the study ..o definitely positive, and supports the conclu- 
sion that all three have been attained. Specific conclusions in regard 
to each objective follow. 

Objecti v e No. 1 

Thu procedure for assisting local school district leadership to use 
the evaluation system was determined to be a feasible and general izable 
procedure in nearly all schools involved in the study. Program changes 
have been made and other program changes recommended in the great major- 
ity of these schools concurrently with, or subsequent to local program 
evaluations. The evidence appears to show that the system was also 
effective in changing attitudes and understandings of school faculties 
about vocational education. It was not determined that the system resulted 
in changes in attitudes of citizens who were involved. Local evaluations 
will, in all probability, be conducted by these schools in the future. 

The general staff and citizen reactions to the system or to the activities 
were favorable. 

The objectives of state project plans were quite similar. The 
activities of conducting state projects were similar, in fact quite 
uniform in kind and number. State project plans were adapted somewhat 
to dif fei ences among the states. The activities engaged in reflected 
these differences, but were basically similar. 

Ob jec Lye No, 2 

All states developed, discovered and/or tried out foui' new, or im- 
proved state-level procedures and two tried out five new, or improved 
practices in cooperating schools. These were in addition to those pi ■>- 
ced res originalLy planned and included in memoranda of agreement at 
the start ",f the project. The system, or generalized procedure can be 
improved by addition, or further trial of these practices. 

Objective No. 3 



The development of competencies in evaluation of programs of voca- 
tional education in the state leaders and their assistants, in local 
institutional leadership teams and their cooperating faculties was ac- 
complished. To a probably lesser degree competencies were developed in 
nearly all of the participants of the three regional "dissemination” 
workshops. There was Insufficient evidence available or obtained to 
determine the extent to which competency development reached the poten- 
tial level in those involved. Th* effectiveness of staff and citizen 
involvement in local program evaluation was demonstrated. 



Re commend a t ions 



These recommendations are based on the findings of the present 
study. Their Selection reflects the philosophies of the chief investi- 
gator and ths state leaders involved in the present study. They are 
predicated on the assumption that evaluation of local programs of voca- 
t iona 1 / 1 echn tea 1 education should be conducted by those who are respon- 
sible for and involved in them, and by those who are affected by them. 
They give recognition to the need for statewide program evaluation and 
the respons tb ility for leadership in program evaluation to be provided 
at the state level, 

1, All states that meet the criteria used in this study for select- 
ing etates should try-out the system that was described and tried out 
and demonstrated in the present study. 

a, Leadership for such an effort should be recruited from 
staffs of universities with appropriate leadership resources, of 
the state departments of education and/or of the state research 
coordinating units. 

b, Programs of local leader training in evaluation should be 
provided through regular university courses and workshops, or 
special prepara tory workshops , and through consultant service. 

c, Financial resources should be made available to local 
districts and institutions or agencies involved in worksnops and 
conferences for the extra costs involved, and for the preparation 
of instruments and instructional materials relating to evaluation. 

2. State leadership and resources should be provided for develop- 
ment of evaluative instruments. Example: an instrument for measuring 

faculty find citizen attitudes toward public school programs of vocational 
educa t ion , 



3. A nationwide program of training in local program evaluation 
procedures should be provided, primarily for persons at the state level 
of leadership, but also for administrators and teachers in local, and 
area institutions. The interest of educators in this has been demon- 
strated. Much of the format for such a program has been provided in the 
present project. Ways of linking local program evaluation to statewide 
evaluation would need to be considered. 

4. Further research and development should be conducted on local 
program evaluation ilong several lines* 

a. Study should be given to procedures for relating locally 
directed program evaluations to statewide program and state- 
directed, local program evaluation so that they may complement 
each other and avoid unnecessary duplication of effort. 

b. Further replications of the system should be made to try 
out and demonstrate new or improved practices developed and/or 



tried out in the present study. 

c. More study and development needs to be done on locaL pro- 
cedures for arriving at and stating over-all program objectives, 
particularly with an emphasis on performance, and for developing 
competencies in local administrators and teachers for accomplishing 
this task. 

d. A study should be conducted to determine feasible proce- 
dures, or a system for self-directed evaluation of area vocational/ 
technical school and community college technical programs. 

e. More study needs to be given to the analysis of manpower 
needs to be met by local and area schools beyond those in the im- 
mediate community or district. 
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AN AUDIT OF THE MULTI-STATE PROJECT ON EVALUATION SYSTEMS 



John K. Coster, Director 

Center for Occupational Education 
North Carolina State University at Raleigh 

Introduction 



During the development of the Multi-State Project on Evaluation 
Systems, provisions were made for an audit of the performance of the pro- 
ject in relation to the stated objectives. This section of the report 
presents the results of the audit. 

In order to gather the data for this section of the report, the 
Consultant attended the final session of the state and local leaders in 
each of the four pat ticipat ing states, and the final session of the 
project director and state leaders. State and local reports were re- 
viewed, and the presenters of these reports were questioned, following 
their presentations at the different sessions. Since constraints of 
both time and fiscal resources precluded an in-depth evaluation of either 
the project or its probably longe-range effects on the vocational 
education pr grams in the various schools and states, no pretentious 
claims are being made for the thoroughness of the audit. This section 
of the report is somewhat impressionistic, however, the information which 
has been collected should be both valid and helpful. 

The remainder of this section will discuss not only the actual audit, 
but also turn some attention to the broader problems. of the implications 
of the project from a general systems theory viewpoint and introduce 
some remarks on the general question of self-evaluation. 

Attainment o f the Objectives 

This portion of the audit presents the Consultant’s reactions to 
the extent to which each of the three objectives of the patent project 
were attained. 



Objective 1 . To determine the feasibility and generalizabil i ty 
of a state procedure for assisting local school district leader- 
ship to use the evaluation system developed in Michigan. 



ERIC 



Any attempt to attest to the feasibility and general izability of 
the evaluation system developed in Michigan must be interpreted with 
due regard for the fact that the participating states, and schools within 
the participating states, represented "fixed 11 rather than "random" ele- 
ments. Nevertheless, efforts were made in the selection of states and 
school districts to obtain states containing diverse regions, 
population size, and density. Within the states, the school districts 
were selected on the basis of interest of administrators, location of the 
school district, and, in at least one instance in each state, some unique 
property of the school district. The school districts selected tended to 
be relatively small; no large urban school district was included in the 
O oject. Caution, therefore, dictates that the discussion of feasibility 



- 67 - 



73 



and generalizabili ty be restricted to the four states and twenty school 
districts included in the project. This note of caution does not mean 
that the evaluation system developed in Michigan might not be applicable 
to other states or school districts; rather, it means that the syst.mi was 
not demonstrated in a sample of school districts which would permit 
generallza tlon to a population of school districts. 

By any reasonable set of criteria, the feasibility and general inability 
of the evaluation system developed in Michigan was adequately demonstrated 
in the four states and in the twenty school districts. This is not to 
say that all twenty school districts completed the minimum activities, nor 
that the Michigan system was conformed to equality in each of the school 
districts. However , the variation from the minimum activities was due 
in each instance to circumstances over which the project director, the 
state leaders, and, generally, the local directors had little or no 
control. And if time were to permit, the factors associated with varia- 
tions in effectiveness in carrying out the project in each of the twenty 
schools could be identified and analyzed. The Consultant was impressed 
by the accomplishments of the local project directors, given rather for- 
midable constraints of time and resources. 

Having attested to the attainment of Objective 1, there are a number 
of points that should be made with regard to expectations for extensive 
adoption or adaptation of the Michigan system: 

1. It is doubtful if the objective would have been attained without 
the assistance of the state leaders. The workshops conducted by the state 
leaders, the on-location assistance and supervision, and the constant 

fo] lov-up on problems greatly facilitated the conduct of the project. 

2. It is doubtful if the objective would have been attained withouc 
strong administrative support at the state and local levels. Even tv 1 * 

have been considered as a candidate for a selected state or local district, 
the project director required that letters of interest and support be sub- 
mitted. Furthermore; the remuneration provided by the project to the states 
and to the school districts was re atively modest, and did not reimburs-' 
either the state or the school district for the time incurred in conducting 
the project. Thus, there was ample evidence of strong adminis tr a t ive sup- 
port at both the state and local levels for the conduct of the project. 

3. It is doubtful if Objective I would have been attained without 
a modicum of leadership training covering such aspects of the Michigan 
system as the use of citizen committees and the design and conduct of 
follow-up studies. 

4. The experimental environment or '‘Hawthorne 11 effect was evident, 
and undoubtedly exerted a powerful influence in the conduct ot the project. 
The school districts that were selected were part of a national project, 
and the personnel associated with the project were the recipients of 
special attention from national and state leaders. It may be surmised 
that this effect contributed to the attainment of Objective 1. 

5. There was little or no evidence to Indicate that either the 
size or the type of school organization affected the application of the 
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Michigan system. Nor wa s there evidence to indicate that :he application 
of the system was influenced by the variation among the four states. In 
fact, the differences in the rate and extent of application of the Michigan 
system were greater within states than among states. 

In summary, the Consultant judged that Objective 1 was attained, 
but there is some question of whether the objective would have been 
attained had it not been for the competency of the state leaders, the 
strong administrative support, and the enthusiastic reception and 
involvement of the local personnel. 

Objective 2. To discover and/or develop new or improved procedures 

in ' state system for locai evaluation. 

One of the strengths of the Michigan system is its flexibility, 
and the flexibility of the system was demonstrated in the conduct of the 
project, many of the local directors engaged in a number of activities 
above the required minimum. Generally, these additional activities 
emerged from special interests of the state leaders and local leaders, 
and included an adult-farmer management survey, a placement bureau, a 
study of women out of school and girls in school, and an elementary 
curriculum survey. The approach to Objective 2, however, appeared to 
have been more directed toward catalyzing additional activities within 
the school districts and states than adding to the rationale underlying 
the Michigan system. Thus, the Consultant judged the procedures used 
and activities of the local directors impressive, but did not rate the 
overall attainment of Objective 2 as high as Objective 1. The additions 
appeared to be more in the nature of improvisations on a theme, rather 
than the elaboration of a new theme. 



Objective 3 . To assist in the development of state leadership 
competencies in evaluation of local program of vocational and 
technical education, including creation of the values of local 
involvement. 



The consortium approach used in the project which involved the 
cooperative efforts of the project director and his staff, the state 
leaders from state universities, representatives from the state divisions 
of vocational education, and the local teams was Impressive. The state 
leaders functioned effectively in the conduct of the project within the 
states* There was a healthy interaction between the state leaders and 
the loctil directors and teams. There was evidence of feedback from the 
project into the university base, with such tangible evidences as the 
addition of courses in the evaluation of vocational education at one 
university and a credit course being offered to the local directors at 
another. Although difficult to test, it might be argued that the project 
served as a vehicle for the university-based state leaders to function 
as change agents in the process of the project. The state leaders took 
their task seriously, and there is reason to believe that their knowledge 
of the evaluation process was increased through their participation. 



Although the project has strong administrative support from the 
state directors of vocational education, state personnel were not as 
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heavily involved in the conduct of the project as were the state leaders 
from the cooperating universities. Joint meetings of the consortium pro- 
vided an input into the state staff, but the involvement of che state 
staff was not as extensive. Because of the limited involvement of the 
state staffs in the local projects, the opportunities to develop leader- 
ship competencies through the project were not as great for the profes- 
sional staff in the state divisions of vocational education as for the 
state leaders in the cooperating universities. Further investigation is 
needed to ascertain the extent to which the Michigan system is adopted 
or adapted within the states. The state divisions of vocational education 
were pressing forward toward increased emphasis on evaluation; whether 
there are state resources to provide the leadership needed to implement 
the local system is a moot point at this time. In summary, the project 
provided a basis for the development of state and local leadership per- 
sonnel, in direct proportion to the involvement of these persons in the 
project* and Objective 3 was judged to have been attained to a very 
great extent. 

The Hidden Agenda 

The previous section of this audit gave relatively high marks to 
the attainment of thf. objectives of the Multi-State Project on Evaluation 
Systems. This section of the audit is addressed not to what was heard 
and read, but what might be. The full potential of this project is yet 
to be realized. The potential is implied in the rationale contained in 
the proposal for the project, but it has not been made explicit. 

Underlying the rationale for the project is a concept of total 
community involvement in the improvement of the educational system. This 
underlying concep . infers a systems approach to education in which 
the educational system is conceived as a system within the community and 
where the community acts as the supra-systen for the educational system. 
The community, through its formal and informal leaders sets the goals 
for the system, provides the resources for attaining the goals, sets 
the constraints, and evaluates the outcomes. 

Within the educational system, there is an implied set of subsystems; 
the one of immediate interest being the vocational education subsystem. 
There was interest in linking the vocational education subsystem to 
the other subsystems in the educational system, thereby strengthening 
the vocational education subsystem and embedding the subsystem more 
firmly in the educational system. There was also interest in setting the 
goals of the subsystem, defining the resource requirements, analyzing the 
constraints, and evaluating the output and outcomes. 

Viewed from this frame of reference, the Multi-State Project on 
Evaluation Systems can be conceived as a systems approach to the improve- 
ment of vocational education in the school. This point, however, is not 
made explicit in the manual and was not given currency in the conduct of 
the project. The manual actually describes guidelines and procedures to 
be followed in the process of improving vocational education in the 
schools, and while the approach was systematic the manual docs not 
describe a systems approach. 
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As a point of departure in explicating the application of general 
system theory to the improvement of vocational education in the school 
the role of the advisory committee may be examined. Advisory committees 
have been advocated by vocational educators for years, principally to 
legitimize specific training programs and to gain support for the pro- 
grams, Quite frequently, the advisory committees have been ,f craft M com- 
mittees, which have been appointed to ensure that this program installed 
in the school would prepare persons for entry jobs in an occupational fielc 
The Multi -State Project on Evaluation Systems provided for the constitution 
of a general advisory committee for the total vocational education program 
in the school district, The immediate interest was to generate broad 
support for the vocational program. Under the general system concept, 
the general advisory cotrrmittee provides a basis for linking the vocational 
education subsystem through the educational system to the community supra* 
system. To the extent that the advisory committee represents the formal 
and informal leadership structure in the community then the cormnuni t y be- 
comes more directly involved in the setting of goals, providing resources 
alleviating constraints, and ultimately, with determining the evidence 
required to evaluate the subsystem. 

The hidden agenda in the project, namely that of conceptualizing 
the community as a supra -Syst em, the educational enterprise as a system, 
and the vocational education program as a subsystem, provide? the nucleus 
for the improvement of vocational education in the public schools which 
transcends the stated objectives of the project. Although this conceptu- 
alization is in the embryonic stage, a disservice would be done to the 
improvement of vocational education were this project to stop short of 
realizing its full potential. The full potential o r the project can only 
be realized through the further conceptualizati on of the Michigan project 
cast in a systems framework, which would provide a more adequate basis 
for the training of state and local administrators of vocational education 
Such a program of training would of necessity draw upon general systems 
theory, organizational theory, social organization theory, and evaluatiot 
theory to produce the leadership personnel who would effect cha * -s 1 ti 
the educational system and in the vocational education subsysl' 

A Note of Self Eva lua t ion 

The self evaluation concept embraced in the project is not a new 
concept. The regional accreditation agenciti, for example, are based 
largely on the self evaluation concept. Evaluative standards are set 
by the member institutions, and, while the accrediting agency may be 
thought of as a supra -syst em In general systems terms, the agency derives 
its power and sanctions from the pooled strength of the member Institution: 
Although self-evaluation enterprises are generally more process-oriented 
than produc t -oriented , they nevertheless do possess elements of formative 
evaluation, which provides constant feedback into the process of setting 
goals and objectives and allocating resources to the subsystem. Consider- 
able evidence was presented in the state and local reports to denote that 
the self -examination process led to an improvement in the formative 
aspects of evaluation, although this process was not so identified in 
either the reports, the manual, or in the project proposal. Each local 
Q m reviewed the philosophy and objectives of the vocational programs 
FRIC collected evidence to ascertain whether the goals and objectives hid 




been attained. 



Considerably less attention was given to summative evaluation?. 
Although a number of the participating schools conducted follow-up 
studies, the data generally were restricted to the placement of graduates 
and the return rate in some cases was not sufficiently large to justify 
the conclusions that frequently were drawn* Further, the objectives 
were stated In broad terms, which were not amenable to the evaluation of 
the output of programs. In addition, since the follow-up instruments 
were constructed locally, their quality varied widely among the schools. 
If the project evidenced one serious fault, it probally was that the 
activities of the participating schools exceeded reasonable expectations 
in terms of the resources-- t ime and funds--ava ilable to the local project 
directors, with the result that soma activities such as the construction 
of follow-up instruments did not receive the careful attention which 
might be desired. Given the benefit of hindsight and the relatively 
secure position of the consultant, it might be argued that the benefits 
of a standardized follow-up instrument and a standardised set of in- 
structions, carefully pre-tested, vould have offset the possible advan- 
tage of local involvement in the constr ction of such instruments. 
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University, (1966) 58 pp. 

Stock, Wm. E. , & Pratzner, Frank C., Review of Research on Student 
Selection and the JP redlcation of Success in Occupational Educa - 
t ion , Minneapolis, Minnesota, Minnesota R.C.U. University of 
Minnesota, (1969) 40 pp. 

The Advisory Committee and Vocational Educatio n, Washington, D#C. 

American Vocational Association (1969) 52 pp. 
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Vocational -Technical Program Evaluation , (An Instrument) (Form 753-14 
4-70) St. Paul, Minnesota, Minnesota Department of Education, 

Voc. Tech. Division, Centennial Bldg. (1970) 4 pp. 

Vocational-Technical School Evaluation Criteria , Bureau of Curriculum 

Development and Evaluation, p.O. Box 911, Harrisburg, Pennsylvania, 
Division of Evaluation, Pennsylvania Department of Education, (197°) 

80 pp. 

Ward, Charles F. , National Conference on Accreditation of Public Po st- 
secondary Occupational Education , Center of Conference Report No. 

11, Raleigh, N.C. Center for Occupational Education, North Caro- 
lina State University, (1970) 78 pp. 

Wenr i h , Ralph C . , Review and Synthesis ' f Res earch on the Administra - 
tion Of Vocational and Technical Edu jtion , ERIC Clearinghouse 
on Vocational and Technical Education, 1900 Kenny Rd . , Columbus, 

Ohio, The Center For Vocational and Technical Educ., The Ohio 
State University, (1970) 56 pp. 

Wisconsin Department of Public Instruction. "A Manual to Be Used in the 
Evaluation of Thirty-Four Comprehensive High Schools in Wisconsin 
Which Participated in a Three-Year Pilot Program of High School 
Vocational Education/ 1 Madison, Wisconsin, (1968) 100 pp. 
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SELECTED RESEARCH REPORTS ON ERIC MICROFISCHE 



ED 016077 


Using Benefit - Cost Analysis in Planning and Evaluating 
Vocational Education. Davie, Bruce F. Nov. 65 1 20 pp. 


ED 016083 


The Development and Testing of an Evaluation Model For 
Vocational Pilot Programs. Final Report Tuckman, Bruce 
W. Rutgers, The State Univ., New Brunswick, N.J. Jul. 67* 
104 pp. OEG- 1-7^0083 55 -*2039 


ED 016767 


The Measurement of Vocational Needs. Minnesota Studies In 
Vocational Rehabilitation. Number 16. Weiss, David J. 
and others, Minnesota Univ., Minneapolis, Ind. Relat. 
Center Bull. 39 Apr. 64 1 104 pp. 


ED 016812 


Implications of Women's Work Patterns for Program Develop- 
ment in Vocational and Technical Education. Lee, Sylvia 
L. and others, Ohio State Univ. Columbus, Center for 
Voc. Educ. Res.-Ser.-19 Oct. 6 7 1 80 pp . 


ED 016852 


A Proposed Model For An Information Storage and Retrieval 
System For Reporting Job Placement Follow Through Data Of 
Persons Trained In Industrial Education Programs In 
California Public Schools. Tentative Draft. Harris, 
Wayne M- California State Dept, of Education, Sacramento 
June 67' 44 pp* 


ED 016852 


Development and Administration of the Youth Opinion Question- 
naire. Minnesota Studies in Work Attitudes, Tech. Report 
No. 2. Grean, George B., Davis, Rene V. Minnesota Univ. 
Minneapolis, Oct. 66* 52 pp. 


ED 021981 


Handbook For Estimation of Labor Supply By Use Of A Community 
Survey. Palomba, Iowa S5ate Manpower Development Council, 

Des Moines. Feb. 67* 29 pp. 


ED 022046 


Predictors of Success In H*gh School Level Vocational Educa- 
tion Programs. A Review, Predigar, Dale J. and others. 
Spons. Agency-Office of Education (DHEW) , Washington, D.C. 
Dec. 67' 32pp. 
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Appendix A 



Members of Advisory Committee to the Project, and Consultants Used 
in Meetings and Workshops 

Advisory Committee 

George Ekstrorc, Professor Emeritus, University of Missouri 
George Ferns, Professor, Michigan State University 

Peter G. Haines, Professor, Business and Distributive Education, Michigan 
State University 

Vanetta Kell, College of Business Education, Northern Arizona University 
William Pierce, formerly Deputy State Director of Voc . Education, Michi- 
gan Department of Education 

Dennis Roley, Director of Business & Office Occupations, Washington 
Department of Education 

Harold Star r, Center for Research and Leadership Development in Vocational 
and Technical Education, Ohio State University 
Richard Warren, Superintendent, Niles, Michigan Public Schools 
Ralph Wenrich, Professor, Department of Vocational Education and Practical 
Arts, University of Michigan 

Consultants 

Lawrence Borosage, Professor, Industrial Education, Michigan State Univer- 
sity 

Gerald Butts, Director of Vocational Education, Corunna, Michigan 
John K. Coster, Director, Center for Occupational Education, Research- 
Development -Training , North Carolina State University, Raleigh 
Edward Crawford, Planning and Development Branch, Division of Vocational - 
Technical Education, U.S. Office of Education, Washington, D.C. 
William Davis, Vocational Couselor, Niles, Michigan Public Schools, Niles, 
Michigan 

Max U. Eninger, President, Educational Systems Research Institute, Inc. 

Pi t tburgh 

Peter Haines, Professor, Business and Distributive Education, Michigan 
State University 

Arthur J, Jones, Jr., Professor, Mississippi State University 
Russell Maples, Director of Vocational Education, Lansing, Michigan, 

Public Schools 

Edgar Persons, Professor, University of Minnesota 
David Pucell, College of Education, University of Minnesota 
Edward Remick, Consultant in Research, Lansing, Michigan Public Schools 
John Rolloff, Professor of Vocational Education, University of Arkansas 
Brandon Smith, College of Education, University of Minnesota 
Glenn Smith, Vocational Division, Michigan Department of Education 
Dan Stevens, Director of Vocational Education, Niles Michigan Public 
Schools 

H. Paul Sweany, Professor, Michigan State University 
Richard Warren, Superintendent, Niles Michigat Public Schools 
Kenneth Wilkinson, Professor, Mississippi Stete University 
J. S. Wlttman, Jr., Professor, Mississippi State University 

o 
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Appendix B 





Workshop for State Project Leaders 
Michigan State University 
Multi-State Project in Evaluation Systems 

March 18, 19, 20, 1968 

Agenda 


Monday 


Rm. 510 Erickson Hall 


9:00 


-Overview, and reorientation to the project 
-Plans for the workshop 

-Selection of cooperating schools, local leadership teams, 
and projection of activities 


10:00 

10:15 


Break 

-The role of the administrator in local program evaluation 
Mr. Richard Warren, Superintendent of schools, Niles* Michigan 
-The role of local leaders and associates 


11:45 

12:00 

1:30 


Scheduling individual conferences on state project plans 
Lunch 

Procedures in assessment of outcomes of local programs, 
determining local program objectives and behavioral goals 


3:15 


Rm, 507, Erickson Hall 


4:00 

8:00 


The processes of local staff involvement 
Individual conferences on state project plans 
Individual conferences on state project plans 


Tuesday 


Rm. 507, Erickson Hall 


8:30 


The processes of citizen involvement in evaluation - Dr. Kenneth 
Wilkinson, Associate Social Psychologist, Social Science Research 
Center, Mississippi State University 


9:45 

10:00 


Break 

Using citizen'. 1 advisory committees in local program evaluation ~ 
Dan Stevens, Director of Vocational Education, Niles, Michigan 


1.2:00 


Luncheon - Kellogg Center - Michigami Room Dr. Carl Gross and 
Dr, Peter Haines, Resource Persons 


1:30 


Procedures in conducting local follow-up studies - Resource 
persons : 

Gerald Butts, Director of Vocational Education, Corunna, 
Michigan, and 

Glenn Smith, Supervisor, Operations, Vocational Education, 
M,D,E . 


3:00 

3:15 


Break 

Evaluation of local projects, state projects, and records and 
reports 


4:00 

7:30 


Individual conferences on state project plans 
Individual conferences on state project plans 
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Agenda con't. 



Wednesday Inn America - Parlor C 

8:30 Development of plans for workshops for local project leader- 
ship teams 
10:00 Break 

10:15 Development of schedules of state activities for the next 9 

months - State project leaderj 

Development of schedule of project leader's activities - 
Byram 

12:00 Adjournment 
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Appendix C 



SUBCONTRACT 

between 

MICHIGAN STATE UNIVERSITY 
and 

UNIVERSITY OF ARKANSAS 



WHEREAS Michigan State University is the recipient of a grant from the 
U.S. Office of Education, Grant No. OEG-0-070968-2329 for a re- 
search project entitled n A Five-State Try-out and Demonstration 
Program to Determine the Generalizability of an Evaluation Sys- 
tem. for local Programs of Vocational and Technical Education/ 1 
and 

WHEREAS the scope of work provides for subcontracting tor research and 
development assistance at state and local levels, 



NOW THEREFORE, both parties to this contract agree to carry out the acti- 
vities as specified in the scope of work incorporated herein by 
reference and attached hereto as Appendix A, 

I Consideration : The Michigan State University agrees to reimburse the 

University of Arkansas for actual expenses incurred for this 
project not to exceed $4,000. Reimbursement shall be monthly or 
quarterly as agreed to between the parties of this subcontract. 

II Period : The period of this subcontract is from February 1, 1968 

through January 31, 1969 

III Reports : Subcontractor shall submit one copy of quarterly technical 

reports and a final report within 30 days of the expiration date. 

IV General Provisions : 

A. The Grant Terms and Conditions for Research Grants, U.S. Office 
of Education are hereby made a part of this agreement and are 
attached as Appendix B. 

B. Administration of the professional functions will be under the 
direction of the Michigan State University Research and Develop- 
ment Program in Vocational-Technical Education. 

C. All research data and reports are confidential and the property 
of the United States Office of Education until formally released 
by the Michigan State University Project Director in conformity 
with the terms of the U.S. Office of Education contract. 



V Project Leader : Subcontract 

Norton 

For: Michigan State University 

East Tensing, Michigan 

BY: 

Jack Breslin. Secretary 
Board of Trustees 



Date: 
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3-20-68 



Project leader shall be Dr. Robert 

For: University of Arkansas 

BY: 

Authorized Representative 
J.R, Carney, Asst. Secretary 
Board oi Trustees 

Date; 3-15-68 
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APPENDIX A 



Scope of Work 

1. A program of orientation and preparation of state leaders will be 
provided, using resources of the Michigan State University Research 

and Development Program in Vocational and Technical Education, and con- 
sultants from outside the university. 

2. The principal investigator (Michigan project leader) will also serve 

in a consultant capacity to the state project leaders, and will assist, 
as appropriate, in state conferences and workshops for training local 
leaders in their role in local program evaluation. The manual on evalua- 
tion of local programs will be made available to state and local leaders, 
as well as local staff evaluation committee members as soon as possible. 

3. Each state project will, in general, follow the same procedures as 
are being followed in the current Michigan Evaluation Systems pro- 
ject. The state leaders will each select five schools having programs 
of vocational education in three or more fields. Each leader will be 
expected to submit a project plan. The elements of this plan will include 
the following: 



A. Statement of the purpose 

B. Progressive steps to be followed in the state developmental pro- 
ject. 

1) Activities or responsibilities of the state leader. These 

activities will include the following categories: selection of cooper- 

ating schools; orientation of local administrators and project leaders 
and research associates in these schools; development and completion of 

an agreement with each school; provision of leadership development through 
a workshop and conferences; provision of consultant services; reviewing 
progress of each school; obtaining and analyzing local records; and pre- 
paring state reports. 

2) The minimum activities to be required in each local school 

will be stated in the local agreement with the ^ca te agency or institution. 
These are: 0) a working committee of the faculty, consisting of a 

teacher representing each vocational field in the program, a represent- 
ative of the guidance staff, and other teachrrr and assistant administra- 
tors; (2) a citizens* advisory committee to work with each local leader 
and staff conmittee; (3) a follow-up survey of former students; and (4) a 
statement of philosophy and objectives of the local program of vocational 
and technical education. States may add to these minima if they wish. 
Activities that states might wish to suggest to local schools would be: 
program descriptions for evaluation purposes; study of needs for vocational 
education programs; evaluation of the practical aits program in relation 

to vocational education; feasibility studies of shared time and other 
cooperative arrangements; and other activities that are deemed appropriate 
by state project leaders. 
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C. Schedule of activities (beginning and ending or completion dates) 

4. Each state will require reports from cooperating schools, and will 
prepare periodic reports and final report. 

5. Periodic reports and a final report will be prepared by the chief 
investigator . 

6. The chief investigator (Michigan project leader) and each state 
project leader will cooperatively appraise each state project, with 
the help of qualified consultants. 

7. The manual on local program evaluation developed in the pioneer 
evaluation project will be revised in the light of experiences and 
findings of the proposed project. 
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Appendix D 



PROGRESS REPORT MEETING 
FOR STATE PROJECT LEADERS 
MULTI -STATE PROJECT ON EVALUATION SYSTEMS 

December 7, 1968 
Parlor 750 
Adolphus Hotel 

AGENDA 

8:30 Reviews of developments and status of projects in each state 
by state leaders (V hour per state) 

10:00 Break 

10:15 Continuation of state reviews 

10:45 Questions and discussion relative to state reports 

11:00 Analyzing follow-up data in relation to criterion questions 

Dr. Max U. Eninger, resource person* 

12:00 Recess for lunch 

1:30 Securing information for determining the attainment of project 
obj ec tives 

2:15 Contract and budget for next year 
2:45 Break 

3:00 Agenda for state project review meetings, spring and fall 

3:30 Future schedules - state and MSU project leaders 

3:45 Logistics, announcements and appointments 

4:00 Adjournment 

★President, Educational Systems Research Institute, Pittsburgh, Penn- 
sylvania 
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Appendix E 



MULTT -STATE PROJECT ON EVALUATION SYSTEMS 

State Project Leaders Conference 

Graduate School of Education Building 

University of Arkansas 

June 2 - 3, 1969 

Monday, June 2 
Room 115 

Reports of progress, status, concerns, and plans of state projects 
8:30 a.m. The Nevada project - Howard Christensen 

9:30 Break 

10:00 a.m. The Mississippi project - James Wall 

11:00 a.ro. The Minnesota project - Paul Marvin 



12:00 



Lunch 



1:30 p.m. Presentation of a representative Arkansas local 

project by local leadership team 

2:30 p.m. The Arkansas project - Robert Norcon 

3;30 p.m. Break 

4:00 Tour of the building 

6:00 p.m. Dinner - Downtown Motor Lodge 

Dr. Henry Kronenberg, Dean of Education, Chairman 

"Vocational Education in Arkansas", J. Marion Adams, State Director of 

Vocational Education 

"The Program of Vocational Teacher Education of the University" 

Dr. Denver Hutson, Head, Department of Vocational Education. 
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Tuesday, June 3 
Room 115 



Forenoon 



8:30 




Plans for evaluation of local and state projects and 
the evaluation system 

. - The design for determining attainment of objectives 

- Depth interviews 

- Check list for staffs and citizens 

- Other instruments and sources of evidence 


9:30 




Break 


10:00 




Relevance of 0VIS to local program evaluation 


10:30 




Role of state leaders in assistance to local leaders 
in the implementation phase of the project 


11:00 




Development of format, and plans for assembling iufor 
mation for final state reports 






Lunch 


1:00 




Suggestions for revision of manual 


2:00 




Fall local project leaders* meetings, purposes, 
agenda, consultants, dates 


2:30 


* 


Break 


3:00 




Planning final meeting of state leaders - purposes, 
agenda, dates 


o 

o 

■c 




Proposed leadership development workshops for 1970 


5:00 




Adjourn 
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Appendix F 



MULTI-STATE PROJECT ON EVALUATION 
OF LOCAL PROGRAMS OF VOCATIONAL EDUCATION 

MEETING OF STATE PROJECT LEADERS 
St. Louis, Missouri, January 13-14, 1970 

Holiday Inn-North 
Living Room 



Objectives of the Meeting 



1, To cooperatively review reeults of the four state projects 

2. To cooperatively review and evaluate the system of locally directed 
e valuat ion 

3, To identify improvements possible in the system *nd/or suggestions 
that might be included in the manual 

4, To retrieve ar iences , data and information pertinent to develop- 
ment of the final report on the project 

5. To summarize dissemination activities to date and to plan for further 
dissemination of information 

6. To review proposal and obtain suggestions regarding the three regional 
workshops to be conducted 



AGENDA 

T uesday 



8:30 State reports 

Twenty-minute summaries and review of state and local project 
activities and outcomes 

10:00 Break 

10:13 Reports continued 

11:00 Accomplishments on M-S Project Objective No. 2 - "New, or Improved 
Procedures for State Assisted, Locally Directed Evaluations." 

11:45 Lunch 

1:00 Summary and conclusions on Project Objective No. 1 (See mimeo: 
"Design for Determining the Attainment of Objective No, 1") 

3:00 Break 

ERJC 
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3:15 


Reports on special aspects 

-Junior college evaluation - Mississippi 

-County unit evaluation - Mississippi, Nevada 

-Area school evaluation - Minnesota 

-VEII analysis - Arkansas 

-OVIS - all 

-Dissemination - all 


4:30 


Final report - completion of contract commitments 


5:00 


Recess and dinner 


7:00 


Time for completion of any items remaining in preceding agenda 
and/or for individual conferences and/or recreation 

Wednesday 


8:30 


Gener alizability of the strategy for leadership and assistance to 
locally directed evaluation 

-Difficulties and problems encountered and not solved 
-Weaknesses in the strategy and the system 
-Strengths 


9:30 


Summary and Evaluation - John K, Coster 


10:00 


Break 


10:15 


Review of manuscript for Manual on Evaluation (third edition) 
Discussion of plans for publication 


12:15 


Lunch 



f^O^^lans and suggestions for furtttCI: dissemination of the strategy 





and the system, its results and leadership development 


2:00 


Planning for the three regional workshops 

-Plans anc. decisions made to date - Revised prospectus 
-Objectives, expected outcomes 

-Recruitment and selection of participants - Brochure 
-Topics for presentations 


3:00 


Break 


3:15 


Selection of presenters and discussion leaders 
Evaluation of the workshop 
Daily schedule 

Pre-planning conferences to be held 

logistics 

Other 


5:00 


Adjournment 
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Appendix G 



Suggested Outlines For State Leader and 
Local Leader Workshop Participants to Use 
in Developing Plans 
•kitick'k'k 

Plan for Directing an Evaluation of a Local or Area 
Program in Vocational-Technical Education 

Name of school or college 

Address of school or college 

Name of author of the plan/person 

who will be the leader of the project ___ 

A. The Situation 

Describe the local or area situation, administrative commitment to local 
program evaluation and the responsibility you have, or will have in 
directing the evaluation effort. 

B. Purposes or Objectives of the Project 

State here the purposes of the local evaluation effort 

C. Activities to Be Undertaken 

Include here the activities to be undertaken by the local leadership 
team, by local evaluation committee or staff committee, and by other 
members of the faculty in accomplishing the purposes stated under B, 
including plans for involvement of citizens' committees and staff. 

D. Time Schedule of Activities 

►* ^ 

Place in calendar sequence the activities listed in C. 

E. Leader and Staff Time Needed 

F. Consultant Services Needed 

Give purposes and nature of consultant service needed, and possible 
sources of such assistance. 

G. References and Instructional Resources Needed for Staff and 
Advisory Committees 

H. Kinds of Records to be Kept 

Include records of activities, meetings, data collected for evaluation, 
and others. 



I. Reporting to the State 

Tndicate what will be reported to the State Department of Education 



J. Financial Needs 



Estimate needs for financial support for activities such as: clerical 

assistance, supplies, consultant help, data processing, reporting, 
communications, travel, etc. 

****** 

Plan for Provision of State-Level Assistance and Leadership Training 
For Persons Who Will Direct/Conduct Evaluations of Local or Area 
Programs of Vocational/Technical Education 

Name of author of this plan 

Position held by author of this plan 

A. Situation 

Describe the kinds of schools and/or community colleges to be served; the 
type of commitment you have; responsibility you have or expect to 
have with regard to training local or area leaders; assistance to these 
schools or other Institutions; and/or criteria to be used to select/ 
identify Institutions to be assisted. 

B. Training to Be Provided to Local or Area Leaders 

List specific activities in each area of training for local or area 
evaluation below: 

a. Activities for stating philosophy, objectives, and criterion 
ques t ions 

b. Activities for involving staff and citizens 

c. Studying the existing local or area programs and needs of 
employers, students, others 

d. Activities for identifying and obtaining evidences of attain- 
ment of local or area program objectives; data and other 
information, instruments, surveys, interviews, etc. 

e. Activities for analyzing data, formulating and implementing 
recommendations 

C. Local Leadership Teams 

List the criteria to be recommended to local administrators for selection 
of persons for local leadership teams. 

D. Local Commitment 

How much time will be expected of local leadership team and staff, and 
how will commitments to this and to the total local evaluation effort 
be obtained? 



E. Involvement 

Nature and amount of staff and citizen involvement that will be expected 
in the cooperating schools. 
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F* Consultant Services 



Indicate functions to be performed by consultants who have had expedience 
in evaluation. 



G. Organization 

Place in calendar sequence the activities in the training program 

H. Financial Needs for the Training or Assistance Program 
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DAILY DISCUSSION EVALUATION FOR 

Please circle the group you most appropriately fit into; 

A, Area School Leader C. State Dept. Leader 

B. Local School Leader D. University /R. C.U. Leader 

PART I - Presentations 



Read each statement carefully. Decide how you react to each state- 
ment and circle the most appropriate answer: (A=agree; U=undecided; and 

D=disagree) Please briefly elaborate on items where needed. (see Example) 



STATEMENTS 


PRESENTATION 

RATINGS 


QUALIFYING BRIEF COMMENTS 








Example : 








Byrara - Short 




0 . 


Presentations were 














appropriate in length 


A 


U 


D 


Ferguson - too wordy 




1 . 


Presentations were 


A 


U 


D 








valuable to me* 












2. 


Presentations were pert- 














inent to the workshop 














objectives. ^ 


A 


u 


D 


% 


- 


3. 


Presentations were 














appropriate in length. 


A 


u 


D 






4. 


levels of complexity 














or difficulty of presen- 














tations were appropriate 


A 


u 


D 






3. 


Visuals (were/would be) 














aids to understanding 


A 


u 


D 






6. 


Visuals were of suitable 














quality. 


A 


u 


D 






Others : 


A 


u 


D 










A 


u 


D 
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PART II - Group-Work Sessions 



Please read each statement carefully, and circle the answer most 
nearly describing your reaction. Use the space below each statement 
for brief elaboration oh your answers. (A=agree ; U-undecided; and 
D=d is agree) 



STATEMENTS 

1. Purposes for this session were clear to rne* 

2. Group leaders provided needed assistance. 

3. Our discussion was appropriately held to 
the purposes. 

4. My questions and comments were encouraged. 

5. Widespread interaction among participants 
was helpful to me. 

6. Differing opinions didn't disrupt group 
cohesiveness . 

7. I was stimulated to think objectively. 

8. Purposes for this session were achieved. 



Responses 
A U D 

A U D 

ADD 
A U D 

A U D 

A U D 

A U D 

T U D 



Others : 



3 
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LOCAL PROGRAM EVALUATION WORKSHOP 



FINAL EVALUATION 



Please circle the group of which you are a part: 

A. Area School B, Local school C. State Dept. D. Univ/R.C*U. 

List two or three portions of the workshop which you consider to have 
been the "high lights” for you. 

1 . 

2 . 

3. 

You probably had some specific objectives for attending, and there 
were four objectives for this workshop as perceived by the staff. Your 
opinions are desired, as a portion of the evaluation of this workshop. 
Your suggestions and comments will be considered in planning the last 
in this series of workshops. Please circle the most appropriate answers 
to the following, writing in brief elaborations on those of special 
concern to you. 



EXAMPLE Agre e Und ec Ided 

0, This workshop was highly structured: <£T u 

, "Rctf cO 



Disagree 

D 



1. This workshop sensitized me to issues 

involved in plannin^and conducting ^ 
local program evaluations. A 

2. This workshop identified the types of 

dara, and information upon which 
programs might be evaluated A 

3. Sources of available materials and 

information on evalurtion were 
identified. A 

4 # Methods of securing required evaluation 

data and information were identified* A 

3. The information presented was valuable 

to me. A 

6. Illustrative materials, and/or examples 
of suggested procedures have been help- 
ful to me. A 




U 



U 



u 

u 

u 



u 



D 



D 



D 

D 

D 



D 
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Undecided 



Pis agree 



7. Interaction between state, local, and 

area leaders was helpful to me* A 

8. The project staff's purposes for this 

workshop were clear to me . A 

9. The objectives of the leaders for this 

workshop were not the same as my 
objectives for attendance* A 

10. I didn't have ray objectives met. A 

11. Possible solutions to my problems were 

covered, A 



U 

U 



U 

U 

U 



D 

D 



D 

D 

D 



12, The information presented was too 
advanced . 



A 



U 



D 



13. 


The information presented was too 
elementary. 


A 


U 


14. 


The discussion leaders were not well 
prepared 


A 


U 


15. 


I was stimulated to think objectively, 
about the topics presented. 


A 


u 


16. 


We did not relate theory to practice. 


A 


u 


17. 


The sessions followed a logical 
sequence , 


A 


u 


18. 


The schedule was too fixed or rigid* 


A 


u 


19. 


I did not have opportunities to 
express my opinions* 


A 


u 


20. 


I was included as a contributing 
participant at this workshop. 


A 


u 


21. 


My time attending this workshop 
was well spent . 


A 


u 


22. 


The workshop lived up to ray 
expectations . 


A 


u 


23. 


1 have no adequate guides for future 
a ct ion . 


A 


u 



D 

D 

D 

D 

D 

D 

D 

D 

D 

D 

D 



24. The research findings presented will be 

useful to me on my job. A 

ERIC 
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Agree Undecided 



Disagree 



25. The references available to partici- 
pants were not appropriate, A U 

26. The video-tape contributed to my 
understanding of how local evaluation 

may be organized and conducted. A U 

27. Other visuals used were helpful to my 

understanding of the topics presented. A U 

28. The manual should have been given 

out and read prior to presentation. A U 

29. More time should have been provided 
for r 

a. Discussions. A U 

b. Consultations while 

developing proposals. A U 



30. State leaders should have been paired 

with local/area leaders, and vice-versa, 
as consultants in plan proposal develop- 
ment. A U 

31. Workshops of this nature should be: 

a. Offered again in future 

years to selected persons. A U 

b. Offered to all interested 
Voc. Ed. leaders at appro- 
priate locations for involving 

various groups. A U 



D 



D 

D 

D 



D 

D 



D 



D 



D 



List other changes in the workshop which would have made it more beneficial 
to you: 
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Appendix J 



Outline for Final Report of local Project 
Recommended by a State Project Leader 

MISSISSIPPI EVALUATION SYSTEMS 
PROJECT IN OCCUPATIONAL EDUCATION 

FINAL REPORT FORM 



I . ABSTRACT 



Occupies one or two pages, identifies the school, and concisely and 
simply summarizes the objectives, findings, conclusions, etc. 



I I . INTRODUCTORY SECTION 



A. Introduction 

1, The school - This division gives background information 
about the school system. Such items as size, district 
served, enrollment, faculty, etc. may be included. 

2. Vocational Program - This division gives a description of 
the school’s vocational program, which may include state- 
ments on the school’s vocational education philosophy and 
objectives . 

B. Objectives of Local School Evaluation 

This division gives a clear, concise listing of objectives 
established within the framework of the evaluation project 
by each school. These objectives give direction to each 
school's evaluation effort. 

C. Plans for Evaluation 



This division records the plans made for the evaluation effort. 

It may include a list of proposed activities, minutes from 
planning meetings, plans for committees selection and use of 
consultants, etc. 

D. Description of Activities 

In this division a detailed description is given of each activity 
included in the evaluation project. A time schedule and other 
materials may also be included. 

E. Staff Involvement 



o 
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This division presents information focusing upon the extent 
ard type of staff involvement. It reports the outcomes, attitudes, 
type committees, functions, philosophy changes, successes, 
failures, etc. 
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F. Citizen Involvement 

Facts about citizen involvement in evaluation are presented 
in this division. It reports types, size and representation of 
committees, functions, attitudes, outcomes, successes, failures, 
etc , 

G. Consultant Involvement in Froject 

This division presents information focusing upon the role 
consultants played in the evaluation project. 

H. Public Relations 

This division deals with the type, method, and scope of public 
relations used in connection with the evaluation project. 

III. F INDINGS SECTION 

A. Committees Activities 

This division reports the number, type, and scope of all 
activities of each committee. 

B. Committees Recommendations 

The recommendations of each committee are presented in this 
divi si on . 

C. Follow-up Studies 

If follow-up studies of graduates, dropouts, etc. were made, 
the results, generalizations regarding successes and failures 
are reported in this division. 

D. Curricular Studies 

If studies of present or projected curricula were made, the 
results are reported in this division. 

E. Other Types of Studies 

If occupational surveys, attitude studies, or other types 
of studies were made, the results are reported in this division. 

F. Changes Resulting from or Related to Local Program Evaluation 

Any changes, innovation, adoption, or plans resulting from the 
project are reported in this division. 

IV, CONCLUSIONS. RECOMMENDATIONS, AND IMPLICATIONS 

i . Conclusions 



o 

ERIC 



- 99 - 



lUo 



The conclusions are organized according to objectives that were 
drawn up for the project in each school system. 



B, Recommendations and Implications 

All recommendations, implications, and future plans for evaluation 
should be reported in this division. 

V. APPENDIX 

A. Agreements 

B. Questionnaires 

C. Check lists 

D. Committee membership lists 

E. Public relations materials 

F. Etc. 
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Appendix K 



Copy of Program of One Dissemination 
Workshop 



LOCALLY DIRECTED EVALUATION OF PROGRAMS OF VOCATIONAL EDUCATION: 
A REGIONAL WORKSHOP FOR STATE > LOCAL, AND AREA LEADER? 



Memphis, Tennessee 
Holiday Iun> Brooks Rd. 



November 3-6» 1970 



Sponsored by 
The Multi -State Project 



College of Education 
Michigan State University 
and Cooperating States 
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LOCALLY DIRECTED EVALUATION OF PROGRAMS OF VOCATIONAL EDUCATION: 

A regional workshop for state, local, and area leaders 

Sponsored By 

The Multi -State Project on Local Program Evaluation 
Memphis, Tennessee 
November 3-6, 1970 



Purposes of the Workshop 

1. To assist state educational leaders in planning programs for pre- 
paring local and area leaders in directing and conducting local pro- 
gram evaluation. 

2. To prepare local or area educational leaders to direct and conduct 
local program evaluations. 

3. To disseminate and review information about program evaluation gained 
in the Multi-State Evaluation Project* 



Outcomes 



Each participant is expected to develop a tentative plan of action which 
has relevance to his work at home base. These plans will be reviewed 
and discussed with other participants at the workshop. Plans will be 
devaloped as follows: 

a. State leaders will develop plans for training and assisting 
local or area leaders in directing local program evaluation. 

b. Local or area leaders will develop plans for evaluating pro- 
grams of vocational and/or technical education in their 
institutions . 
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8:00 

8:30 



10:00 

10:30 



12:00 



0 




Tuesday* November 3 



A.M. 



Conference registration, Banquet Rm. 
No. 2 

First General Session 



Welcome, introductions, and work- 
shop arrangements 

Purposes, outcomes, use of manual 
and references 

11:1-5* Locally Directed Evaluations: The 

Rationale, Essential Elements and 
Steps In Organizing and Directing 
Them 



Question period 



Plans for workshop evaluation 



Break 

First session, continued 

A Local Program Evaluation at 
Cross ett, Arkansas 



Question period 



The Role of Administration in Local 
Program Evaluation 



Lunch 

★Refer to section in Manual 



James Wall 
Presiding 

James Wall 



Robert Norton 



Harold Byram 



Gordon 

Ferguson 



Mrs. Bernice 
Crawford 
and 

Mrs. Francis 
Richards 



Ed Hayman 
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Tuesday, P.M . 



1:30 



First General Session* continued 



IV: 1 -10 
VII: 3 -4 

App . G 



3:00 



Formulating Statements of Philo- 
sophy and Objectives, Identifying 
Sources and Obtaining Evidences of 
Attainment - A presentation and 
discussion 

Guide Lines for Preparation of 
Evaluation Plan - Kentucky 

The Charge to Workshop Participants 



Break 



3:13 



4:30 



First Group -Work Sessions 

Group Leader 

A James Wall 

B Gordon Ferguson 

C Harold Byram 

D Robert Norton 



Recess, staff meeting 



7 : 00 - 9:00 



Individual work on plans 



Meecings by states - optional 
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Harold Byram 
Presiding 



James Wall 



Charles Neel 
Robert Norton 



Resource Persons 

Mrs. Crawford 
Mrs. Richards 
Charles Neel 
Ed Hayman 
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Wednesday, November 4 



8:30 



10: 15 
10:30 

12:00 



0 




A.M. 



Second General Session Harold Byram 

presiding 

111:1-2, How Does a Iocal/Area Leader Team 

8-11 Develop a Plan and Direct the Robert Norton 

App, B Local Evaluation Effort 



Question period 



How Does a State Leader Develop a 

Plan, and Work With the Training James Wall 

of Local/Area Evaluation Leaders? 



Question period 



Kentucky Project for Locally 

Directed Evaluation Charles N;el 



Break 



Second Group-Work Sessions 



Group 


Leader 


Resource Persoi 


A 


James Wall 


Charles Neel 


B 


Gordon Ferguson 


Gerald Butts 


C 


Harold Byram 




D 


Robert Norton 




Lunch 
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Wednesday, P.M . 



1:00 

2:45 



4:30 

5:30- 



O 




First Individual Work Session 



Consultants: Gerald Putts, Harold 

Byrara, Gordon Ferguson, Charles 
Neel, Robert Norton, & James Wall 



Third General Session: Robert Norton 

Presiding 

Involving Citizens in Local 
Evaluations 

111:3-8 Principles and Procedures, Emphasis 

App . A on Advisory Committees 

How We Worked with Citizens 1 
Committees at Corunna, Michigan 

Question period 



Recess, staff meeting 



:00 Trip to Tennessee State Technical 

Institute, arrangements to be 
announced 

Video-tape showing of Blooming Prairie 
Minn, team report 



Harold Byram 



Gerald Butts 
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Thursday, November 5 



A.M. 



8:30 



Fourth General Session Janes Wall 

Presiding 



V Planning and Conducting Follow 

App. C, D up 



Key Points to be Observed James Shill 



How We Conducted and Used a 

Follow-up Survey at Corunna, Gerald Butts 

Michigan 



Question period 



10:00 



Break 



10:30 



Third Group-Work Sessions 

Group Leader 

A James Wall 

B Gordon Ferguson 

C Harold Byrain 

D Robert Norton 



Resource 

Charles Neel 
Gerald Butts 

James Shill 



12 s 00 



Lunch 
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Thursday, P.M. 



1:30 



Fifth General Session 



VI Studying the Program in Relation 

to Needs 



App. E, H How Do We Determine if Program 
Meets Manpower Needs? 



App. F, G How Do We Determine if Program 

Meets Needs and Concerns of Students 
and Parents? 



Question Period 



3:00 



Break 



3; 15 Second Individual Work Sessions “ 

Completion of writing plans (Plan 
readers available as consultants) 



4:30 Staff Meeting 



7:00-9:00 Scheduled reading of plans, and 

individual conferences 

Plan readers and consultants: 

Gerald Butts L/A 
Harold Byram S/U 

Gordon Ferguson L/A 
Charles Neel S/U 
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Harold Byram 
Presiding 



James Wall 



Robert Norton 



Robert Norton S/U 
Marvin Robertson 
L/A 

James Shill S/U 
James Wall S/U 



Friday, No\ ember 6 
A.M. 



8:00 

9:00 



10:00 

10:30 

12:00 

1:30 



2:30 

2:45 

3:00 

4 : 00 




Individual conferences on plans, cont. 

Fifth General Session Harold Byram 

Pres iding 

VII Interpreting Findings, Making James Shill 

VIII: 1-3 Recommendations and Reporting 

App. K, I, 

J, K 



How Should a Local or Area Team Gordon Ferguson 

Evaluate Its Own Evaluation Project? 



Break 



Relating Local Evaluations to State 

Wide Evaluations Robert Norton 



Planning Ahead in States Represented James Wall 
Lunch 



VIII :4 



£JL. 

Fourth Group-Work Sessions 



Sharing of information on plans and 
their implementation 



Group Leader 

1 Harold Byram 

2 James Wall 

3 Robert Norton 

4 Gordon Ferguson 



Resource Persons 



James Shil) 



Sixth General Session 



James Wall 
Presiding 



Travel reimbursement and work- 
shop evaluation forms Gordon Ferguson 

A Look at the Future in Evalua- 
tion Harold Byram 

What Have We Done and What Are We 
Going to Do? Panel discussion: 
four participants and James Wall, 
panel chairman 



Adjournment 
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WORKSHOP STAFF 



Co -L-i rector s : 



Harold Byram, Professor of Education, Michigan State University 

Ja.es Wall, Assistant Dean, Research and Development, Mississippi 
State University 

Multi-State Project Staff : 

Robert Norton, Assistant Professor of Vocational Education, 
University of Arkansas 

James Shill, Co-Director, Research Coordinating Unit, Vocational- 
Technical Education, Mississippi State University 

Gordon Ferguson, Multi-State Project Assistant, Michigan State 
University 

Associated Staff : 

Gerald Butts, Director of Vocational Education, Corunna, Michigan 
Mrs. Bernice Crawford, Crossett Public Schools, Crossett , Arkansas 
Ed. Dayman, Superintendent of Schools, Amory, Mississippi 
Mrs. Frances Richards, Crossett Public Schools, Crossett, Arkansas 
Participant - Consultant : 

Charles 0. Neel, Coordinator, State Program Evaluation, State 
Department of Education, Frankfort, Kentucky 
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Appendix L 

LOCALLY DIRECTED EVALUATION 
OF 

PROGRAMS OF VOCATIONAL EDUCATION 



HOW-TO WORKSHOPS 




Sponsored by 
The Matte-State PAojec* 

College of Education 
Michigan State University 
and Cooperating States 




t 




The work presented or reported 
herein was performed pursuant 'jo 
a grant from the U.S. Office of 
Education, Department of Health, 
Education, and Welfare. 
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Appendix M 



Multi-Stale Project 
on Eialootioo Systems 




A Systems 



Appioach to the 
[valuation of 
Vocational [ducation 




Research and Development Program 
in Vocational-Technical Education 



Department of Secondary Education and Curriculum 
College of Education 
MICHIGAN STATE UNIVERSITY 

October 1968 
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Multi-Stale Pioject 
on [valuation Systems 

A part of the background for this and the 
previous projects in Michigan is the mani- 
festation of an increasing interest in 
evaluation of vocational and technical edu- 
cation. Larger dollar investments in edu- 
cation, strong demand by employers for 
highly skilled and technically trained 
workers, unemployment and underemployment 
of a portion of the labor force, and the 
attention being focused on the importance 
of each individual being given the oppor- 
tunity to develop to his fullest potential 
are factors contributing to the increased 
interest in evaluation. 

While teachers in many schools have con- 
ducted evaluations of their own instruction 
in various subject areas, few schools have 
taken a searching look at TOTAL programs in 
the light of current needs. This is a pri- 
mary goal of this project. 

These factors, combined with the emphasis 
on evaluation in new federal legislation, 
led to the original Michigan project, 1963- 
1965, Three wheels were involved in the 
first project to develoo and try out a sys- 
tematic approach to self-initiated local 
program evaluation. Ten schools were in- 
volved in a further tryout of the system 
during the 1966-1967 academic year. The 
findings of these studies are the basis for 
extending this project to schools in other 
states to determine the general i zabi 1 it y of 
the system. 



Objectives 



The objectives of the project: are: 

©To determine the feasibility and general- 
iz ability of a state procedure for assist- 
ing local school district leadership to 
use the evaluation system developed in 
Michigan . 

•To discover and/or develop new or improved 
procedures in a state system for local 
evaluations . 

•To assist in development of state and 
local leadership competencies in evalu- 
ation of loral programs of vocational and 
technical education, including creation 
of understanding of the values of local 
involvement . 



Assumptions 



There are several assumptions underlying 
this and the previous projects conducted 
in Michigan. 

•Evaluation should be done by those respon- 
sible for line program and those affected 
by it. 



• r mphasis in program evaluation in local 
public schools should be primarily on 
goals and outcor.es. Modern philosophy 
of evaluation gives strong support to 
the goal s-out comes approach. The ques- 
tion suggested is : ,r How well is the 

school achieving the goals of prepara- 
tion for the world of work?' 1 
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•Evaluations conducted in the past, in which 
consultant help has been used, often have 
emphasized ways and means rather than goals 
and outcomes. Professional educators now, 
however, are seeking ways of evaluating 
total contributions of the school to occu- 
pational preparation. They are also con- 
vinced that provisions for evaluation need 
to be built into future plans for vocation- 
al education in the public schools. 



The Systems 
Approach 



This project, with its emphasis on goals 
and outcomes, calls for the application of 
the systems approach. The system involves 
the definition of desired outcomes as a 
first planning step, followed by assessment 
of the outputs and then an analysis of 
available and needed resources to more effi- 
ciently achieve those outcomes. 



Theoretically at least, every school staff 
member -- but particularly those who have 
responsibilities for specialized occupa- 
tionil preparation — should be involved in 
the total evaluation project. State leader- 
ship teams and other consultants are used 
to help local school personnel in their 
evaluation off oris, but actual implementa- 
t ion and operation of the program is in the 
hands nf the school's administration and 
staff. 

The project operates on a consortium basis, 
with contribution^ made by the V. S. Office 
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of Education, Michigan State University, the 
participating states, and local schools. 

Local schools are encouraged to develop and 
utilize an evaluation system most suitable 
to their needs. Certain minimum activities 
are performed in each school, including the 
followf ng ; 

•The development and/or refinement of a 
statement of philosophy and objectives of 
the local program of vocational and tech- 
n i c a 1 educa t ion. 

•The establishment of a working committee 
of t. e faculty, consisting of a teacher 
representing each vocational field in the 
program, a representative of the guidance 
staff, and other teachers and administra- 
t or s . 

•The establishment of a citizens’ advisory 
committee to work with each local leader 
and staff committee. 

•The Completion of a follow-up study of 
former students. 



Responsibilities 

of Panicipanis mutiica:: state vmiyfksity. 



• Prevision of c-anfer^nrpo vork. i-y q to 

fuitJid develop c\ ** i ' ■ t. ii.*i eevp'. t [ ’iu’ i ■ s 
represent .at i ves fit- par i if i t it \ .states. 
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•Consultative service* on evaluation of vo- 
cational education. 

•Evaluation manual, instruments and related 
materials . 

•Provision of consultants for state work- 
shops . 



•Visits by project leader to coopeiating 
states . 

•Research assistance personnel: (1) to 

state project leaders, and (2) to local 
project leaders. 

•Publication of the final report of the 
proj ect . 



PARTICIPATING STATES: 



•Provision of services of state project 
leader for 1968 and 1969. 

•Clerical services and facilities. 

•Leadership and consultation to local pro- 
ject leaders, 



•Conferences and workshops to furtlicr 
develop evaluation competencies of local 
leaders . 

•Analysis of records, reports, and other 
data fron cooperating schools, 
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•Publication and dissemination of reports 
on the project. 



Participating 
States in the 
Multi State 



COOPERATING SCHOOLS: 



•Initiation and operation of an evaluation 
project in line with generp.l guidelines 
provided . 

•Services of local leadership team (usually 
a project director and research associate) 
coiTjn tins urate with the size and scope of 
the vocational education program. 

•Clerical services and facilities. 

•Appropriate record keeping and preparation 
of report on the local project . 



• ARKANSAS 



Ptojeci 



• '-'JNNTSOTA 



•MISSISSIPPI 



•NKVAHA 
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Dr* Robert Norton 




STATE PROJECT TFADER 
Dr. Robert. Norton 



Vnca L ii'iicil Education Dopar Indent 

l n i v or s i t y o f A r ka n s a s 

Pa yo 1 1 tv i lie, Arkansa s 7 2 7 0 1 




ASSISTANT STATE PROJECT TRADER 
Nr. George Brookor 
Vocational Education Department 
University of Arkansas 
Kaye 1 1 uv i 1 1 e , Arkansa s 727 0 I 




PARTI Cl EAT INC SCHOOLS AND 1 OPAL PROJECT LEADERS 

1. CROSSETT PL- BMC SCHOOLS , CROSS). n 

Mrs. Bernice Crawford, Of lice Lducat ion Tear Lev 

2. HARRISON PUBLIC SCHOOLS , 1 ’ARM SUN 
Mr . V. H. MuCutcheon, Ceunpelnr 

3. ROGERS PUBLIC SCHOOLS, ROGERS 

Mr. J. W, Crimes, Vocational Education Teacher 

4. RUSSE1 TV 1 1 IE PC BMC SCHOOLS, KLSSLI.LVULE 

Mr* Bill 1.. lewis, Distributive Education Coordinator 

3. TEXARKANA PUBLIC SCHOOLS, TEXARKANA 

Mr, Javos D. Kytl.er, Vocational Education Pirector 

b. NEWPORT PUBLIC SCHOOLS, NEWPORT 
Mr. William D, Shelton, Counselor 

lad 
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Dr. R. Paul Marvin 




MINNESOTA 



STAIR PROJECT TRADER 
Mr. R. Paul Marvin 
20S T.or t ic u 1 1 ur c- Du i Id i ng 
University of Minnesota 
St. Paul, Minnesota 55101 



ASSISTANT STATE PROJECT TRADE 
Mr. Edward Hartog 
208 Horticulture Building 
University of Minnesota 
St. Paul, Minnesota 55101 



PART 1C [PAT TNG SCHOOLS AND LOCAL PROJECT LEADERS 

1. ANOKA -HENNEPIN SCHOOL DISTRICT iUl , ANOKA 
Mr. Robert Grose, Vocational Coordinator 

2. BLOOMING PRAIRIE PUBIIC SCHOOLS, B [.COMING 

PRAIRIE 

Mr. Wallace Wilke, Vocational 'Teacher 

3. ChNiENNJAL SCHOOL DISTRICT ^>12, CIRCLE PINES 
Mr. Carlo Sooger, Special Programs 



HUTCHINSON PUBLIC SCHOOLS, HUTCHINSON 
Jounettc Gynan N Mary lhoehl, Vocational 

STAPLES AREA VOCA'l TONAL -TTCHNICAL SCHOOL 
Mr. Stanley A. Ldiii, Assistant Diicctor 



F JT aclier s 
, STAN ES 
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MISSISSIPPI 



STATE PROJECT LEADER 
Dr. Jar.es E. Wall 
Social Science Research Center 
Director K.C.J. 

Mississippi State University 
State College, Mississippi 39762 



ASSISTANT STATE PROJECT LEADER 
Dr. Jnres F. Shill 
Social Science Research Center 
Mississippi State University 
State College, Mississippi 39762 




I'AUf IMPATIT.C SMimis AND LOCAL V*niJKCT LEADERS 

1. AMORY SCHOOLS, AMOKY 

. Mabry Allison, Coordinator 
V< v a k i r f i a 1 Of T r cc Tr a i i: i n g 

7. i.Oi U'AK Ujl/NIY SCHOOLS, Cf.hVl.l.AND 

• Ot*i>»ge T. Dovol 1 , Vocational Agriculture Teacher 

1. JACKSON CMl'Ni V JUNIOR C<>U LOL, CAU1 TER 
Mi. havis Ferguson, Coord in itor 

A, : ;r COKi; nj'MV SUI'HUS, dkeknwood 

I-t . Chw'ies r. Peyo, Vocational Education Director 

3. 1'AFf ADO"! A CI1Y SCH001S, PASCAGOULA 

Mt . Ro’vo !>. LuVe, Assistant Superintendent 



•: r 



NEVADA 
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static project lfadlr 

M r . Hci w.i r d Cor d L s r o 
Heroes Memorial Building 
Nevada State Department of 
Education 

Carson City, Nevada 89701 



Mr . Ed va I'd Cord i sro 




ASSISTANT STATU PROTECT LEAD id 
M r . LVnn’is j r a h ar , 

Heroes Memorial Building 
Nevada State Department of 
Euii c a t i on 

Carson City, Nevada 89701 



PARTICIPATING SCHOOLS AND I. 0 CA 1 PROJECT LEADERS 

3, CHURCHILL COUNTY SCHOOL DISTRICT, FAIT ON 
Mr. Elliot Lina 



ELKO COUNTY SCHOOL DISTRICT, hi KO 
Mr. Don hirer 

0 RMS BY COUNTY SCHOOL D 1 S 1 RK T, CARSON tTTY 
Mr, ihriy DicLson 

WASIIOK COl N ! Y SCUnOI, DISTRICT, RENO 
Mr. Ceoige Brighton 

Kill IE I'lNL COUNT Y SCHOOL DISIRICl. KEY 
Mr. Burl on iovoisui 



Id. 
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Relation to R S El 



Headquarters for the Multi -State Evaluation 
Systems Project is the Research and Develop- 
ment Program in Vocational-Technical Educa- 
tion at Michigan State University. The R & D 
Program Is based on the clinical approach. 
I.ocal schools and school systems are in- 
volved as partners in the testing and try- 
out phases of theoretical constructs for 
curriculum, administrative patterns, teacher 
education, instructional materials devel- 
opment and other aspects of vocat ional- tech- 
nical education. 



MSP Project Leader 



n. 




Dr. Harold M . Bvi an 
Professoi of Education 
J39 Erickson Hal! 

Michigan State University 
East L-msing, Mich. 4882 3 
IMione: 5 l 7-33'* - 1837 



MSU Assistant Project Leader 




Floyd L. McKinney 
Doctoral Candidate in 
f "due a l ion 
240 Ei iokson Hill 
Michigan State University 
Ur. s t L an s { n g „ M i c h . 4882 J 



The research and evaluation reported herein was performed 
pursuant to a contract with the United States Department 
of Health, Education and Welfare, Office of Education. 
TT c O.E. Project Number 7-0968. Grant Nurcbev O.E.G.-0-9- 
68 > 




